SampledRankBasedEvaluator

class SampledRankBasedEvaluator(evaluation_factory, *, additional_filter_triples=None, num_negatives=None, head_negatives=None, tail_negatives=None, **kwargs)[source]

Bases: RankBasedEvaluator

A rank-based evaluator using sampled negatives instead of all negatives.

See also [teru2020].

Notice that this evaluator yields optimistic estimations of the metrics evaluated on all entities, cf. https://arxiv.org/abs/2106.06935.

Initialize the evaluator.

Parameters
  • evaluation_factory (CoreTriplesFactory) – the factory with evaluation triples

  • additional_filter_triples (Union[None, LongTensor, List[LongTensor]]) – additional true triples to use for filtering; only relevant if not explicit negatives are given. cf. pykeen.evaluation.rank_based_evaluator.sample_negatives()

  • num_negatives (Optional[int]) – the number of negatives to sample; only relevant if not explicit negatives are given. cf. pykeen.evaluation.rank_based_evaluator.sample_negatives()

  • head_negatives (Optional[LongTensor]) – shape: (num_triples, num_negatives) the entity IDs of negative samples for head prediction for each evaluation triple

  • tail_negatives (Optional[LongTensor]) – shape: (num_triples, num_negatives) the entity IDs of negative samples for tail prediction for each evaluation triple

  • kwargs – additional keyword-based arguments passed to pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator.__init__()

Raises

ValueError – if only a single side’s negatives are given, or the negatives are in wrong shape

Methods Summary

evaluate_ogb(model, mapped_triples[, batch_size])

Evaluate a model using OGB's evaluator.

process_scores_(hrt_batch, target, scores[, ...])

Process a batch of triples with their computed scores for all entities.

Methods Documentation

evaluate_ogb(model, mapped_triples, batch_size=None, **kwargs)[source]

Evaluate a model using OGB’s evaluator.

Parameters
  • model – the model; will be set to evaluation mode.

  • mapped_triples (LongTensor) –

    the evaluation triples

    Note

    the evaluation triples have to match with the stored explicit negatives

  • batch_size (Optional[int]) – the batch size

  • kwargs – additional keyword-based parameters passed to pykeen.nn.Model.predict()

Return type

MetricResults

Returns

the evaluation results

process_scores_(hrt_batch, target, scores, true_scores=None, dense_positive_mask=None)[source]

Process a batch of triples with their computed scores for all entities.

Parameters
  • hrt_batch (LongTensor) – shape: (batch_size, 3)

  • target (Literal[‘head’, ‘relation’, ‘tail’]) – the prediction target

  • scores (FloatTensor) – shape: (batch_size, num_entities)

  • true_scores (Optional[FloatTensor]) – shape: (batch_size, 1)

  • dense_positive_mask (Optional[FloatTensor]) – shape: (batch_size, num_entities) An optional binary (0/1) tensor indicating other true entities.

Return type

None