Metrics

A module for PyKEEN ranking and classification metrics.

Classes

Metric()

A base class for metrics.

ValueRange([lower, lower_inclusive, upper, ...])

A value range description.

RankBasedMetric()

A base class for rank-based metrics.

ClassificationMetric()

A base class for classification metrics.

Class Inheritance Diagram

digraph inheritance3ebc3345fc { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "ClassificationMetric" [URL="../api/pykeen.metrics.ClassificationMetric.html#pykeen.metrics.ClassificationMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base class for classification metrics."]; "Metric" -> "ClassificationMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ABC" -> "ClassificationMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ExtraReprMixin" [URL="utils.html#pykeen.utils.ExtraReprMixin",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A mixin for modules with hierarchical `extra_repr`."]; "Metric" [URL="../api/pykeen.metrics.Metric.html#pykeen.metrics.Metric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base class for metrics."]; "ExtraReprMixin" -> "Metric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "RankBasedMetric" [URL="../api/pykeen.metrics.ranking.RankBasedMetric.html#pykeen.metrics.ranking.RankBasedMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base class for rank-based metrics."]; "Metric" -> "RankBasedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ValueRange" [URL="../api/pykeen.metrics.ValueRange.html#pykeen.metrics.ValueRange",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A value range description."]; }

Ranking metrics.

This module comprises various rank-based metrics, which get an array of individual ranks as input, as summarize them into a single-figure metric measuring different aspects of ranking performance.

We can generally distinguish:

Base Metrics

These metrics directly operate on the ranks:

The following metrics measures summarize the central tendency of ranks

The Hits at K metric is closely related to information retrieval and measures the fraction of times when the correct result is in the top-\(k\) ranked entries, i.e., the rank is at most \(k\)

The next metrics summarize the dispersion of ranks

and finally there is a simple metric to store the number of ranks which where aggregated

Inverse Metrics

The inverse metrics are reciprocals of the central tendency measures. They offer the advantage of having a fixed value range of \((0, 1]\), with a known optimal value of \(1\):

Adjusted Metrics

Adjusted metrics build upon base metrics, but adjust them for chance, cf. [berrendorf2020] and [hoyt2022]. All adjusted metrics derive from pykeen.metrics.ranking.DerivedRankBasedMetric and, for a given evaluation set, are affine transformations of the base metric with dataset-dependent, but fixed transformation constants. Thus, they can also be computed when the model predictions are not available anymore, but the evaluation set is known.

Expectation-Normalized Metrics

These metrics divide the metric by its expected value under random ordering. Thus, their expected value is always 1 irrespective of the evaluation set. They derive from pykeen.metrics.ranking.ExpectationNormalizedMetric, and there is currently only a single implementation:

Re-indexed Metrics

Re-indexed metrics subtract the expected value, and then normalize the optimal value to be 1. Thus, their expected value under random ordering is 0, their optimal value is 1, and larger values indicate better results. The classes derive from pykeen.metrics.ranking.ReindexedMetric, and the following implementations are available:

z-Adjusted Metrics

The final type of adjusted metrics uses the expected value as well as the variance of the metric under random ordering to normalize the metrics similar to z-score normalization. The z-score normalized metrics have an expected value of 0, and a variance of 1, and positive values indicate better results. While their value range is unbound, it can be interpreted through the lens of the inverse cumulative density function of the standard Gaussian distribution to retrieve a p-value. The classes derive from pykeen.metrics.ranking.ZMetric, and the following implementations are available:

Functions

generate_ranks(num_candidates[, ...])

Generate random ranks from a given array of the number of candidates for each ranking task.

generate_num_candidates_and_ranks(num_ranks, ...)

Generate random number of candidates, and coherent ranks.

generalized_harmonic_numbers(n[, p])

Calculate the generalized harmonic numbers from 1 to n (both inclusive).

harmonic_variances(n)

Pre-calculate variances of inverse rank distributions.

Classes

RankBasedMetric()

A base class for rank-based metrics.

DerivedRankBasedMetric([base_cls])

A derived rank-based metric.

ExpectationNormalizedMetric([base_cls])

An adjustment to create an expectation-normalized metric.

ReindexedMetric([base_cls])

A mixin to create an expectation normalized metric with max of 1 and expectation of 0.

ZMetric([base_cls])

A z-score adjusted metrics.

ArithmeticMeanRank()

The (arithmetic) mean rank.

AdjustedArithmeticMeanRank([base_cls])

The adjusted arithmetic mean rank (AMR).

AdjustedArithmeticMeanRankIndex([base_cls])

The adjusted arithmetic mean rank index (AMRI).

ZArithmeticMeanRank([base_cls])

The z-scored arithmetic mean rank.

InverseArithmeticMeanRank()

The inverse arithmetic mean rank.

GeometricMeanRank()

The (weighted) geometric mean rank.

AdjustedGeometricMeanRankIndex([base_cls])

The adjusted geometric mean rank index (AGMRI).

ZGeometricMeanRank([base_cls])

The z geometric mean rank (zGMR).

InverseGeometricMeanRank()

The inverse geometric mean rank.

HarmonicMeanRank()

The harmonic mean rank.

InverseHarmonicMeanRank()

The inverse harmonic mean rank.

AdjustedInverseHarmonicMeanRank([base_cls])

The adjusted MRR index.

ZInverseHarmonicMeanRank([base_cls])

The z-inverse harmonic mean rank (ZIHMR).

MedianRank()

The median rank.

InverseMedianRank()

The inverse median rank.

HitsAtK([k])

The Hits @ k.

AdjustedHitsAtK([base_cls])

The adjusted Hits at K (\(AH_k\)).

ZHitsAtK([base_cls])

The z-scored hits at k (\(ZAH_k\)).

StandardDeviation()

The ranks' standard deviation.

Variance()

The ranks' variance.

Count()

The ranks' count.

NoClosedFormError

The metric does not provide a closed-form implementation for the requested operation.

AffineTransformationParameters([scale, offset])

The parameters of an affine transformation.

Class Inheritance Diagram

digraph inheritance03f5cf17f0 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AdjustedArithmeticMeanRank" [URL="../api/pykeen.metrics.ranking.AdjustedArithmeticMeanRank.html#pykeen.metrics.ranking.AdjustedArithmeticMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The adjusted arithmetic mean rank (AMR)."]; "ExpectationNormalizedMetric" -> "AdjustedArithmeticMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AdjustedArithmeticMeanRankIndex" [URL="../api/pykeen.metrics.ranking.AdjustedArithmeticMeanRankIndex.html#pykeen.metrics.ranking.AdjustedArithmeticMeanRankIndex",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The adjusted arithmetic mean rank index (AMRI)."]; "ReindexedMetric" -> "AdjustedArithmeticMeanRankIndex" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AdjustedGeometricMeanRankIndex" [URL="../api/pykeen.metrics.ranking.AdjustedGeometricMeanRankIndex.html#pykeen.metrics.ranking.AdjustedGeometricMeanRankIndex",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The adjusted geometric mean rank index (AGMRI)."]; "ReindexedMetric" -> "AdjustedGeometricMeanRankIndex" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AdjustedHitsAtK" [URL="../api/pykeen.metrics.ranking.AdjustedHitsAtK.html#pykeen.metrics.ranking.AdjustedHitsAtK",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The adjusted Hits at K ($AH_k$)."]; "ReindexedMetric" -> "AdjustedHitsAtK" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AdjustedInverseHarmonicMeanRank" [URL="../api/pykeen.metrics.ranking.AdjustedInverseHarmonicMeanRank.html#pykeen.metrics.ranking.AdjustedInverseHarmonicMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The adjusted MRR index."]; "ReindexedMetric" -> "AdjustedInverseHarmonicMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AffineTransformationParameters" [URL="../api/pykeen.metrics.ranking.AffineTransformationParameters.html#pykeen.metrics.ranking.AffineTransformationParameters",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The parameters of an affine transformation."]; "ArithmeticMeanRank" [URL="../api/pykeen.metrics.ranking.ArithmeticMeanRank.html#pykeen.metrics.ranking.ArithmeticMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The (arithmetic) mean rank."]; "RankBasedMetric" -> "ArithmeticMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Count" [URL="../api/pykeen.metrics.ranking.Count.html#pykeen.metrics.ranking.Count",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The ranks' count."]; "RankBasedMetric" -> "Count" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DerivedRankBasedMetric" [URL="../api/pykeen.metrics.ranking.DerivedRankBasedMetric.html#pykeen.metrics.ranking.DerivedRankBasedMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A derived rank-based metric."]; "RankBasedMetric" -> "DerivedRankBasedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ABC" -> "DerivedRankBasedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ExpectationNormalizedMetric" [URL="../api/pykeen.metrics.ranking.ExpectationNormalizedMetric.html#pykeen.metrics.ranking.ExpectationNormalizedMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An adjustment to create an expectation-normalized metric."]; "DerivedRankBasedMetric" -> "ExpectationNormalizedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ExtraReprMixin" [URL="utils.html#pykeen.utils.ExtraReprMixin",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A mixin for modules with hierarchical `extra_repr`."]; "GeometricMeanRank" [URL="../api/pykeen.metrics.ranking.GeometricMeanRank.html#pykeen.metrics.ranking.GeometricMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The (weighted) geometric mean rank."]; "RankBasedMetric" -> "GeometricMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "HarmonicMeanRank" [URL="../api/pykeen.metrics.ranking.HarmonicMeanRank.html#pykeen.metrics.ranking.HarmonicMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The harmonic mean rank."]; "RankBasedMetric" -> "HarmonicMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "HitsAtK" [URL="../api/pykeen.metrics.ranking.HitsAtK.html#pykeen.metrics.ranking.HitsAtK",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The Hits @ k."]; "RankBasedMetric" -> "HitsAtK" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InverseArithmeticMeanRank" [URL="../api/pykeen.metrics.ranking.InverseArithmeticMeanRank.html#pykeen.metrics.ranking.InverseArithmeticMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The inverse arithmetic mean rank."]; "RankBasedMetric" -> "InverseArithmeticMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InverseGeometricMeanRank" [URL="../api/pykeen.metrics.ranking.InverseGeometricMeanRank.html#pykeen.metrics.ranking.InverseGeometricMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The inverse geometric mean rank."]; "RankBasedMetric" -> "InverseGeometricMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InverseHarmonicMeanRank" [URL="../api/pykeen.metrics.ranking.InverseHarmonicMeanRank.html#pykeen.metrics.ranking.InverseHarmonicMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The inverse harmonic mean rank."]; "RankBasedMetric" -> "InverseHarmonicMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InverseMedianRank" [URL="../api/pykeen.metrics.ranking.InverseMedianRank.html#pykeen.metrics.ranking.InverseMedianRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The inverse median rank."]; "RankBasedMetric" -> "InverseMedianRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MedianRank" [URL="../api/pykeen.metrics.ranking.MedianRank.html#pykeen.metrics.ranking.MedianRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The median rank."]; "RankBasedMetric" -> "MedianRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Metric" [URL="../api/pykeen.metrics.Metric.html#pykeen.metrics.Metric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base class for metrics."]; "ExtraReprMixin" -> "Metric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "NoClosedFormError" [URL="../api/pykeen.metrics.ranking.NoClosedFormError.html#pykeen.metrics.ranking.NoClosedFormError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The metric does not provide a closed-form implementation for the requested operation."]; "RankBasedMetric" [URL="../api/pykeen.metrics.ranking.RankBasedMetric.html#pykeen.metrics.ranking.RankBasedMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base class for rank-based metrics."]; "Metric" -> "RankBasedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ReindexedMetric" [URL="../api/pykeen.metrics.ranking.ReindexedMetric.html#pykeen.metrics.ranking.ReindexedMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A mixin to create an expectation normalized metric with max of 1 and expectation of 0."]; "DerivedRankBasedMetric" -> "ReindexedMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "StandardDeviation" [URL="../api/pykeen.metrics.ranking.StandardDeviation.html#pykeen.metrics.ranking.StandardDeviation",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The ranks' standard deviation."]; "RankBasedMetric" -> "StandardDeviation" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Variance" [URL="../api/pykeen.metrics.ranking.Variance.html#pykeen.metrics.ranking.Variance",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The ranks' variance."]; "RankBasedMetric" -> "Variance" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ZArithmeticMeanRank" [URL="../api/pykeen.metrics.ranking.ZArithmeticMeanRank.html#pykeen.metrics.ranking.ZArithmeticMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The z-scored arithmetic mean rank."]; "ZMetric" -> "ZArithmeticMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ZGeometricMeanRank" [URL="../api/pykeen.metrics.ranking.ZGeometricMeanRank.html#pykeen.metrics.ranking.ZGeometricMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The z geometric mean rank (zGMR)."]; "ZMetric" -> "ZGeometricMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ZHitsAtK" [URL="../api/pykeen.metrics.ranking.ZHitsAtK.html#pykeen.metrics.ranking.ZHitsAtK",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The z-scored hits at k ($ZAH_k$)."]; "ZMetric" -> "ZHitsAtK" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ZInverseHarmonicMeanRank" [URL="../api/pykeen.metrics.ranking.ZInverseHarmonicMeanRank.html#pykeen.metrics.ranking.ZInverseHarmonicMeanRank",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The z-inverse harmonic mean rank (ZIHMR)."]; "ZMetric" -> "ZInverseHarmonicMeanRank" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ZMetric" [URL="../api/pykeen.metrics.ranking.ZMetric.html#pykeen.metrics.ranking.ZMetric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A z-score adjusted metrics."]; "DerivedRankBasedMetric" -> "ZMetric" [arrowsize=0.5,style="setlinewidth(0.5)"]; }