RankBasedEvaluator

class RankBasedEvaluator(ks=None, filtered=True, **kwargs)[source]

Bases: pykeen.evaluation.evaluator.Evaluator

A rank-based evaluator for KGE models.

Calculates the following metrics:

  • Mean Rank (MR) with range \([1, \infty)\) where closer to 0 is better

  • Adjusted Mean Rank (AMR; [berrendorf2020]) with range \((0, 2)\) where closer to 0 is better

  • Adjusted Mean Rank Index (AMRI; [berrendorf2020]) with range \([-1, 1]\) where closer to 1 is better

  • Mean Reciprocal Rank (MRR) with range \((0, 1]\) where closer to 1 is better

  • Hits @ K with range \([0, 1]\) where closer to 1 is better.

berrendorf2020(1,2)

Berrendorf, et al. (2020) Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods with Adjusted Mean Rank.

Initialize rank-based evaluator.

Parameters
  • ks (Optional[Iterable[Union[int, float]]]) – The values for which to calculate hits@k. Defaults to {1,3,5,10}.

  • filtered (bool) – Whether to use the filtered evaluation protocol. If enabled, ranking another true triple higher than the currently considered one will not decrease the score.

  • kwargs – Additional keyword arguments that are passed to the base class.

Methods Summary

finalize()

Compute the final results, and clear buffers.

process_head_scores_(hrt_batch, true_scores, ...)

Process a batch of triples with their computed head scores for all entities.

process_tail_scores_(hrt_batch, true_scores, ...)

Process a batch of triples with their computed tail scores for all entities.

Methods Documentation

finalize()[source]

Compute the final results, and clear buffers.

Return type

RankBasedMetricResults

process_head_scores_(hrt_batch, true_scores, scores, dense_positive_mask=None)[source]

Process a batch of triples with their computed head scores for all entities.

Parameters
  • hrt_batch (LongTensor) – shape: (batch_size, 3)

  • true_scores (FloatTensor) – shape: (batch_size)

  • scores (FloatTensor) – shape: (batch_size, num_entities)

  • dense_positive_mask (Optional[FloatTensor]) – shape: (batch_size, num_entities) An optional binary (0/1) tensor indicating other true entities.

Return type

None

process_tail_scores_(hrt_batch, true_scores, scores, dense_positive_mask=None)[source]

Process a batch of triples with their computed tail scores for all entities.

Parameters
  • hrt_batch (LongTensor) – shape: (batch_size, 3)

  • true_scores (FloatTensor) – shape: (batch_size)

  • scores (FloatTensor) – shape: (batch_size, num_entities)

  • dense_positive_mask (Optional[FloatTensor]) – shape: (batch_size, num_entities) An optional binary (0/1) tensor indicating other true entities.

Return type

None