Ablation

ablation_pipeline(datasets, models, losses, optimizers, training_loops, *, create_inverse_triples=False, regularizers=None, model_to_model_kwargs=None, model_to_model_kwargs_ranges=None, model_to_trainer_to_training_kwargs=None, model_to_trainer_to_training_kwargs_ranges=None, ablation_config=None, evaluator=None, optuna_config=None, evaluator_kwargs=None, evaluation_kwargs=None, directory=None, dry_run=False, best_replicates=None, save_artifacts=True, move_to_cpu=True, discard_replicates=False)[source]

Generate a set of HPO configurations.

A sample file can be run with``pykeen experiment ablation tests/resources/hpo_complex_nations.json``.

Parameters
  • datasets (Union[str, List[str]]) – A dataset name or list of dataset names

  • models (Union[str, List[str]]) – A model name or list of model names

  • losses (Union[str, List[str]]) – A loss function name or list of loss function names

  • optimizers (Union[str, List[str]]) – An optimizer name or list of optimizer names

  • training_loops (Union[str, List[str]]) – A training loop name or list of training loop names

  • create_inverse_triples (Union[bool, List[bool]]) – Either a boolean for a single entry or a list of booleans

  • regularizers (Union[None, str, List[str]]) – A regularizer name, list of regularizer names, or None if no regularizer is desired. Defaults to None.

  • evaluator (Optional[str]) – The name of the evaluator to be used. Defaults to rank-based evaluator.

  • evaluator_kwargs (Optional[Mapping[str, Any]]) – The keyword arguments passed to the evaluator (in the pipeline)

  • evaluation_kwargs (Optional[Mapping[str, Any]]) – The keyword arguments passed during evaluation (in the pipeline)

  • model_to_model_kwargs (Optional[Mapping[str, Mapping[str, Any]]]) – A mapping from model name to dictionaries of default keyword arguments for the instantiation of that model

  • model_to_model_kwargs_ranges (Optional[Mapping[str, Mapping[str, Any]]]) – A mapping from model name to dictionaries of keyword argument ranges for that model to be used in HPO.

  • model_to_trainer_to_training_kwargs (Optional[Mapping[str, Mapping[str, Mapping[str, Any]]]]) – A mapping from model name to a mapping of trainer name to a mapping of default keyword arguments for the instantiation of that trainer. This is useful becuase for some models, you might want to set the number of epochs differently.

  • model_to_trainer_to_training_kwargs_ranges (Optional[Mapping[str, Mapping[str, Mapping[str, Any]]]]) – A mapping from model name to a mapping of trainer name to a mapping of keyword arguments for that trainer to be used in HPO.

  • ablation_config (Optional[Mapping[str, Mapping[str, Mapping[str, Any]]]]) – Additional third-order and fourth-order ablation configuration for all other ablation keys to models to either kwargs or kwarg ranges

  • optuna_config (Optional[Mapping[str, Any]]) – Configuration passed to optuna for HPO over all ablation studies

  • directory (Optional[str]) – The directory in which the experimental artifacts will be saved.

  • dry_run (bool) – Defines whether only the configurations for the single experiments should be created without running them.

  • best_replicates (Optional[int]) – Defines how often the final model should be re-trained and evaluated based on the best hyper-parameters enabling to measure the variance in performance.

  • save_artifacts (bool) – Defines, whether each trained model sampled during HPO should be saved.

  • move_to_cpu (bool) – Defines, whether a replicate of the best model should be moved to CPU. We recommend to set this flag to ‘True’ to avoid unnecessary GPU usage.

  • discard_replicates (bool) – Defines, whether the best model should be discarded after training and evaluation.

Return type

None

prepare_ablation_from_config(config, directory, save_artifacts)[source]

Prepare a set of ablation study directories.

Parameters
  • config (Mapping[str, Any]) – Dictionary defining the ablation studies.

  • directory (str) – The directory in which the experimental artifacts (including the ablation configurations) will be saved.

  • save_artifacts (bool) – Defines, whether the output directories for the trained models sampled during HPO should be created.

Return type

List[Tuple[str, str]]

Returns

pairs of output directories and HPO config paths inside those directories

prepare_ablation(datasets, models, losses, optimizers, training_loops, *, ablation_config=None, create_inverse_triples=False, regularizers=None, model_to_model_kwargs=None, model_to_model_kwargs_ranges=None, model_to_trainer_to_training_kwargs=None, model_to_trainer_to_training_kwargs_ranges=None, evaluator=None, optuna_config=None, evaluator_kwargs=None, evaluation_kwargs=None, metadata=None, directory=None, save_artifacts=True)[source]

Prepare an ablation directory.

Return type

List[Tuple[str, str]]

Returns

pairs of output directories and HPO config paths inside those directories