Ablation¶
-
ablation_pipeline
(datasets, models, losses, optimizers, training_loops, *, create_inverse_triples=False, regularizers=None, model_to_model_kwargs=None, model_to_model_kwargs_ranges=None, model_to_trainer_to_training_kwargs=None, model_to_trainer_to_training_kwargs_ranges=None, ablation_config=None, evaluator=None, optuna_config=None, evaluator_kwargs=None, evaluation_kwargs=None, directory=None, dry_run=False, best_replicates=None, save_artifacts=True, move_to_cpu=True, discard_replicates=False)[source]¶ Generate a set of HPO configurations.
A sample file can be run with``pykeen experiment ablation tests/resources/hpo_complex_nations.json``.
- Parameters
datasets (
Union
[str
,List
[str
]]) – A dataset name or list of dataset namesmodels (
Union
[str
,List
[str
]]) – A model name or list of model nameslosses (
Union
[str
,List
[str
]]) – A loss function name or list of loss function namesoptimizers (
Union
[str
,List
[str
]]) – An optimizer name or list of optimizer namestraining_loops (
Union
[str
,List
[str
]]) – A training loop name or list of training loop namescreate_inverse_triples (
Union
[bool
,List
[bool
]]) – Either a boolean for a single entry or a list of booleansregularizers (
Union
[None
,str
,List
[str
]]) – A regularizer name, list of regularizer names, or None if no regularizer is desired. Defaults to None.evaluator (
Optional
[str
]) – The name of the evaluator to be used. Defaults to rank-based evaluator.evaluator_kwargs (
Optional
[Mapping
[str
,Any
]]) – The keyword arguments passed to the evaluator (in the pipeline)evaluation_kwargs (
Optional
[Mapping
[str
,Any
]]) – The keyword arguments passed during evaluation (in the pipeline)model_to_model_kwargs (
Optional
[Mapping
[str
,Mapping
[str
,Any
]]]) – A mapping from model name to dictionaries of default keyword arguments for the instantiation of that modelmodel_to_model_kwargs_ranges (
Optional
[Mapping
[str
,Mapping
[str
,Any
]]]) – A mapping from model name to dictionaries of keyword argument ranges for that model to be used in HPO.model_to_trainer_to_training_kwargs (
Optional
[Mapping
[str
,Mapping
[str
,Mapping
[str
,Any
]]]]) – A mapping from model name to a mapping of trainer name to a mapping of default keyword arguments for the instantiation of that trainer. This is useful becuase for some models, you might want to set the number of epochs differently.model_to_trainer_to_training_kwargs_ranges (
Optional
[Mapping
[str
,Mapping
[str
,Mapping
[str
,Any
]]]]) – A mapping from model name to a mapping of trainer name to a mapping of keyword arguments for that trainer to be used in HPO.ablation_config (
Optional
[Mapping
[str
,Mapping
[str
,Mapping
[str
,Any
]]]]) – Additional third-order and fourth-order ablation configuration for all other ablation keys to models to either kwargs or kwarg rangesoptuna_config (
Optional
[Mapping
[str
,Any
]]) – Configuration passed to optuna for HPO over all ablation studiesdirectory (
Optional
[str
]) – The directory in which the experimental artifacts will be saved.dry_run (
bool
) – Defines whether only the configurations for the single experiments should be created without running them.best_replicates (
Optional
[int
]) – Defines how often the final model should be re-trained and evaluated based on the best hyper-parameters enabling to measure the variance in performance.save_artifacts (
bool
) – Defines, whether each trained model sampled during HPO should be saved.move_to_cpu (
bool
) – Defines, whether a replicate of the best model should be moved to CPU. We recommend to set this flag to ‘True’ to avoid unnecessary GPU usage.discard_replicates (
bool
) – Defines, whether the best model should be discarded after training and evaluation.
- Return type
-
prepare_ablation_from_config
(config, directory, save_artifacts)[source]¶ Prepare a set of ablation study directories.
- Parameters
config (
Mapping
[str
,Any
]) – Dictionary defining the ablation studies.directory (
str
) – The directory in which the experimental artifacts (including the ablation configurations) will be saved.save_artifacts (
bool
) – Defines, whether the output directories for the trained models sampled during HPO should be created.
- Return type
- Returns
pairs of output directories and HPO config paths inside those directories
-
prepare_ablation
(datasets, models, losses, optimizers, training_loops, *, ablation_config=None, create_inverse_triples=False, regularizers=None, model_to_model_kwargs=None, model_to_model_kwargs_ranges=None, model_to_trainer_to_training_kwargs=None, model_to_trainer_to_training_kwargs_ranges=None, evaluator=None, optuna_config=None, evaluator_kwargs=None, evaluation_kwargs=None, metadata=None, directory=None, save_artifacts=True)[source]¶ Prepare an ablation directory.