ZHitsAtK
- class ZHitsAtK(base_cls: str | RankBasedMetric | type[RankBasedMetric] | None = None, **kwargs)[source]
Bases:
ZMetric
The z-scored hits at k (\(ZAH_k\)).
Initialize the derived metric.
- Parameters:
base_cls (ClassVar[type[RankBasedMetric] | None]) – the base class, or a hint thereof. If None, use the class-attribute
kwargs – additional keyword-based parameters used to instantiate the base metric
Attributes Summary
whether the metric needs binarized scores
whether there is a closed-form solution of the expectation
whether there is a closed-form solution of the variance
Z-adjusted metrics are formulated to be increasing
Return the key for use in metric result dictionaries.
The name of the metric
whether the metric requires the number of candidates for each ranking task
Z-adjusted metrics can only be applied to realistic ranks
whether the metric supports weights
synonyms for this metric
the value range
Methods Summary
__call__
(ranks[, num_candidates, weights])Evaluate the metric.
adjust
(base_metric_result, num_candidates[, ...])Adjust base metric results based on the number of candidates.
expected_value
(num_candidates[, ...])Compute expected metric value.
Generate the extra repr, cf.
get_coefficients
(num_candidates[, weights])Compute the scaling coefficients.
Get the description.
get_link
()Get the link from the docdata.
Get the math notation for the range of this metric.
get_sampled_values
(num_candidates, num_samples)Calculate the metric on sampled rank arrays.
Iterate over the components of the
extra_repr()
.numeric_expected_value
(**kwargs)Compute expected metric value by summation.
numeric_expected_value_with_ci
(**kwargs)Estimate expected value with confidence intervals.
numeric_variance
(**kwargs)Compute variance by summation.
numeric_variance_with_ci
(**kwargs)Estimate variance with confidence intervals.
std
(num_candidates[, num_samples, weights])Compute the standard deviation.
variance
(num_candidates[, num_samples, weights])Compute variance.
Attributes Documentation
- closed_expectation: ClassVar[bool] = True
whether there is a closed-form solution of the expectation
- key
Return the key for use in metric result dictionaries.
- needs_candidates: ClassVar[bool] = True
whether the metric requires the number of candidates for each ranking task
- supported_rank_types: ClassVar[Collection[Literal['optimistic', 'realistic', 'pessimistic']]] = ('realistic',)
Z-adjusted metrics can only be applied to realistic ranks
- synonyms: ClassVar[Collection[str]] = ('z_hits_at_', 'zahk')
synonyms for this metric
- value_range: ClassVar[ValueRange] = ValueRange(lower=None, lower_inclusive=False, upper=None, upper_inclusive=False)
the value range
Methods Documentation
- __call__(ranks: ndarray, num_candidates: ndarray | None = None, weights: ndarray | None = None) float
Evaluate the metric.
- adjust(base_metric_result: float, num_candidates: ndarray, weights: ndarray | None = None) float
Adjust base metric results based on the number of candidates.
- Parameters:
- Returns:
the adjusted metric
- Return type:
Note
since the adjustment only depends on the number of candidates, but not the ranks of the predictions, this method can also be used to adjust published results without access to the trained models.
- expected_value(num_candidates: ndarray, num_samples: int | None = None, weights: ndarray | None = None, **kwargs) float
Compute expected metric value.
The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters:
num_candidates (ndarray) – the number of candidates for each individual rank computation
num_samples (int | None) – the number of samples to use for simulation, if no closed form expected value is implemented
weights (ndarray | None) – shape: s the weights for the individual ranking tasks
kwargs – additional keyword-based parameters passed to
get_sampled_values()
, if no closed form solution is available
- Returns:
the expected value of this metric
- Raises:
NoClosedFormError – raised if a closed form expectation has not been implemented and no number of samples are given
- Return type:
Note
Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf.
RankBasedMetric.numeric_expected_value()
.
- get_coefficients(num_candidates: ndarray, weights: ndarray | None = None) AffineTransformationParameters
Compute the scaling coefficients.
- Parameters:
- Returns:
a tuple (scale, offset)
- Return type:
- get_sampled_values(num_candidates: ndarray, num_samples: int, weights: ndarray | None = None, generator: Generator | None = None, memory_intense: bool = True) ndarray
Calculate the metric on sampled rank arrays.
- Parameters:
num_candidates (ndarray) – shape: s the number of candidates for each ranking task
num_samples (int) – the number of samples
weights (ndarray | None) – shape: s the weights for the individual ranking tasks
generator (Generator | None) – a random state for reproducibility
memory_intense (bool) – whether to use a more memory-intense, but more time-efficient variant
- Returns:
shape: (num_samples,) the metric evaluated on num_samples sampled rank arrays
- Return type:
- iter_extra_repr() Iterable[str]
Iterate over the components of the
extra_repr()
.This method is typically overridden. A common pattern would be
def iter_extra_repr(self) -> Iterable[str]: yield from super().iter_extra_repr() yield "<key1>=<value1>" yield "<key2>=<value2>"
- Returns:
an iterable over individual components of the
extra_repr()
- Return type:
- numeric_expected_value(**kwargs) float
Compute expected metric value by summation.
The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters:
kwargs – keyword-based parameters passed to
get_sampled_values()
- Returns:
The estimated expected value of this metric
- Return type:
Warning
Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html
- numeric_expected_value_with_ci(**kwargs) ndarray
Estimate expected value with confidence intervals.
- Return type:
- numeric_variance(**kwargs) float
Compute variance by summation.
The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters:
kwargs – keyword-based parameters passed to
get_sampled_values()
- Returns:
The estimated variance of this metric
- Return type:
Warning
Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html
- numeric_variance_with_ci(**kwargs) ndarray
Estimate variance with confidence intervals.
- Return type:
- std(num_candidates: ndarray, num_samples: int | None = None, weights: ndarray | None = None, **kwargs) float
Compute the standard deviation.
- Parameters:
num_candidates (ndarray) – the number of candidates for each individual rank computation
num_samples (int | None) – the number of samples to use for simulation, if no closed form expected value is implemented
weights (ndarray | None) – shape: s the weights for the individual ranking tasks
kwargs – additional keyword-based parameters passed to
variance()
,
- Returns:
The standard deviation (i.e. the square root of the variance) of this metric
- Return type:
For a detailed explanation, cf.
RankBasedMetric.variance()
.
- variance(num_candidates: ndarray, num_samples: int | None = None, weights: ndarray | None = None, **kwargs) float
Compute variance.
The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters:
num_candidates (ndarray) – the number of candidates for each individual rank computation
num_samples (int | None) – the number of samples to use for simulation, if no closed form expected value is implemented
weights (ndarray | None) – shape: s the weights for the individual ranking tasks
kwargs – additional keyword-based parameters passed to
get_sampled_values()
, if no closed form solution is available
- Returns:
The variance of this metric
- Raises:
NoClosedFormError – Raised if a closed form variance has not been implemented and no number of samples are given
- Return type:
Note
Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf.
RankBasedMetric.numeric_variance()
.