Loss Weighting

Loss weighting for triples.

Loss weights influence how much a given triple is weighted in the loss function. They are primarily a tool to shape your optimization criterion, i.e., what you optimize your embedding models on. For example, you may want to focus more or less on certain types of triples because they are more or less important for your application. You can also use loss weights to counteract imbalances in relation frequencies, i.e., to down-weight frequent relation types.

They are not:

1. (static) weights for message passing - we already support those, cf. EdgeWeighting

2. weights in the sense of how reliable a particular triple is. If you have a source of uncertainty about triples, you might want to set lower loss weights for uncertain triples (in the sense of “it does not matter so much if the model reproduces an uncertain label”). Another alternative would be to use softer labels for them (i.e., try to predict the uncertain label directly), or more advanced ways of incorporating uncertainty.

3. more general qualifiers on the triples, e.g., (km, multiple_of, m) could have a qualifier (factor, 10) on it

The current implementation defines the general interface for (dynamic) loss weights via LossWeighter, which you can extend with your own weighting logic. It also provides an implementation of relation-specific weighting, where the loss weight of a triple depends only on the relation type involved: RelationLossWeighter.

Example

Below is a minimal example of how to use it via the high-level pipeline() API:

"""Training with relation-specific loss weights."""

from pykeen.datasets.utils import get_dataset
from pykeen.pipeline import pipeline
from pykeen.triples.weights import RelationLossWeighter

dataset = get_dataset(dataset="CodexSmall")
loss_weighter = RelationLossWeighter.inverse_relation_frequency(mapped_triples=dataset.training.mapped_triples)
result = pipeline(
    dataset=dataset,
    model="MuRE",
    loss="BCEWithLogits",
    training_loop_kwargs=dict(loss_weighter=loss_weighter),
    training_loop="lcwa",
)

Classes

LossWeighter()	Determine loss weights for triples.
RelationLossWeighter(weights)	Determine loss weights based solely on the relation.

Variables

loss_weighter_resolver

A resolver for loss weighters

Class Inheritance Diagram