In applied settings, tests of hypothesis where a nuisance parameter
is only identifiable under the alternative often reduces into one of Testing One
Hypothesis Multiple times (TOHM). Specifically, a fine discretization of the space
of the non-identifiable parameter is specified, and the null hypothesis is tested
against a set of sub-alternative hypothesis, one for each point of the discretization.
The resulting sub-test statistics are then combined to obtain a global p-value.
In this paper, we discuss a computationally efficient inferential tool to perform
TOHM under stringent significance requirements, such as those typically required
in the physical sciences, (e.g., p-value < 10−7
). The resulting procedure leads
to a generalized approach to perform inference under non-standard conditions,
including non-nested models comparisons.