Symposium on Flexible User-Adaptive Multiple Testing Methods - VVSOR - VVSOR

02 December 2026

Symposium on Flexible User-Adaptive Multiple Testing Methods

tiling

We hereby invite you to the symposium “Flexible and User-Adaptive Multiple Testing Methods”. The symposium is organized in honor of the 2025 Hans van Houwelingen Biometry Award won by Jelle Goeman (LUMC) for his paper with P. Górecki, R. Monajemi, X. Chen, T. E. Nichols, W. Weeda: “Cluster extent inference revisited: quantification and localisation of brain activity”, published in Journal of the Royal Statistical Society Series B. (2023).

Speakers: Samuel Davenport (UCSD), Pierre Neuvial (U Toulouse), Jelle Goeman (LUMC), Rianne de Heide (U Twente), Anna Vesely (DKFZ)

Multiple testing is often thought of as a rigid one-off process, to be performed once, after the analysis. Modern multiple testing methods challenge this idea, and instead make multiple testing methods that are flexible and interactive. Using these methods, researchers can revise several of their analysis choices after seeing the data, allowing exploratory data analysis with rigorous error control. This symposium presents some of these new methods, with applications in genomics, neuroimaging and psychometrics.

Location: LUMC, Leiden, Collegezaal 5 (Lecture Room 5, Route 558)

Please register through this link.

Date: Wednesday December 2, 2026, 14:00 – 17:10

 

Schedule:

14:00 14:05 Opening
14:05 14:35 Pierre Neuvial
Flexible false discovery proportion bounds via Joint Error Rate control
14:35 15:05 Rianne de Heide
Dynamic e-closure: Multiple testing when hypotheses and evidence evolve
15:05 15:30 Coffee break
15:30 16:00 Samuel Davenport
16:00 16:30 Anna Vesely
Post-selection Inference in Multiverse Analysis
15:30 17:00 Jelle Goeman
Cluster extent inference revisited: quantification and localisation of brain activity
17:00 17:10 Closing

Abstracts:

Pierre Neuvial

Flexible false discovery proportion bounds via Joint Error Rate control

Modern data analysis, in genomics as in neuroimaging, calls for multiple testing methods that are flexible and user-adaptive: scientists want to explore their data, select and re-select hypotheses after seeing the results, and still obtain valid guarantees on the selected sets.

A general framework for this was proposed by Goeman and Solari (Statistical Science, 2011), who derive simultaneous bounds on the false discovery proportion (FDP) from the closed testing principle. This construction is generic, but it requires dedicated algorithms to be usable in practice. An alternative construction, based on control of the Joint Error Rate (JER), was proposed by Blanchard, Neuvial and Roquain (Annals of Statistics, 2020). It separates the statistical problem, namely achieving JER control that is tight and adapted to the dependence between the tested hypotheses, from the algorithmic one, for which efficient generic solutions are available.
In this talk, I will present this JER-based approach and discuss how to calibrate it to the dependence structure of the data. I will illustrate its use in genomics and neuroimaging, where it provides interpretable bounds on the number of false positives within user-chosen sets of genes or brain regions.

 

Rianne de Heide

Dynamic e-closure: Multiple testing when hypotheses and evidence evolve

In many multiple-testing problems, new hypotheses become available while evidence for existing hypotheses continues to accumulate. A platform trial is a natural example: new treatment arms may open while earlier arms are still recruiting. Standard e-closure applies to a fixed family of hypotheses, but applying it afresh as the family grows can lead to a surprising problem: even adding a new hypothesis with completely neutral evidence can invalidate a previously certified rejection.
In this talk, I will introduce dynamic e-closure, which extends e-closure to growing hypothesis families and anytime-valid evidence. A compatibility condition gives FDR control at arbitrary stopping times. A stronger condition also allows rejections to persist over time, with control of the expected supremum of the false discovery proportion. We establish converse results showing that these closure constructions characterize procedures with the corresponding guarantees.
Finally, I will discuss how to construct coherent intersection e-values. It turns out that coherence across growing hypothesis families imposes strong restrictions on e-value merging. In particular, admissible coherent pointwise mergers under arbitrary dependence must use a single global budget of weights. This leads to efficient procedures, but also a striking impossibility result: symmetric merging rules cannot produce positive evidence on an open-ended hypothesis horizon.

 

Samuel Davenport

Localized Cluster Enhancement: TFCE Revisited with Valid Error Control

Threshold-free Cluster Enhancement (TFCE) is one of the most widely used methods for detecting effects in neuroimaging statistic maps. It was introduced to improve on cluster size inference, with two main claims: that it removes the need to choose a cluster-defining threshold, and that it provides voxel-wise inference with voxel-wise accuracy. In this talk I will argue that the reverse of both claims holds. First, TFCE is not threshold free: its lower integration limit h0 acts as a de facto cluster-defining threshold, and this choice is central to how its results should be interpreted. Second, TFCE controls neither voxelwise nor clusterwise error rates. Its statistic at a null voxel can borrow strength from true signal elsewhere in the brain, so a significant TFCE cluster only implies that there is activation somewhere within a potentially very large support region. I will show, through theory and simulation, that this can lead to severely inflated error rates. To address this, we embed TFCE in a closed testing framework, giving Localized Cluster Enhancement (LCE). LCE recomputes the TFCE statistic on data masked to the region under test and yields adjusted p-values that are valid simultaneously over all regions, including anatomical atlas regions and data-driven clusters chosen after seeing the data, at little extra computational cost. Applied to six task contrasts from the Human Connectome Project, LCE with h0 = 3.1 detects more Harvard-Oxford regions than voxelwise or cluster size inference, while keeping valid error control.

 

Anna Vesely

Post-selection Inference in Multiverse Analysis

With: Paolo Girardi, Daniël Lakens, Gianmarco Altoè, Massimiliano Pastore, Antonio Calcagnì, Livio Finos

When analyzing data, researchers make choices that may be arbitrary, based on subjective beliefs, or for which equally justifiable alternatives exist: how variables are transformed, which confounders are included, which statistical model is used, etc. This wide range of possible choices can be abused and has been identified as one of the underlying causes of the replication crisis. Multiverse analysis evaluates the stability of results across many reasonable specifications, each representing a sequence of data-analytic decisions; however, it is confined to a descriptive role. Specification curve analysis adds an inferential procedure to this framework, but is limited to simple cases within the linear model and only tests the global null hypothesis that no specification has an effect (weak family-wise error rate control). The recently proposed Post-selection Inference for Multiverse Analysis (PIMA) provides a flexible and general inferential approach based on a conditional resampling procedure, accommodating a wide range of data-processing choices and any generalized linear model. The procedure not only tests the global null hypothesis, but also quantifies and identifies specifications with a non-zero effect. In particular, it provides lower confidence bounds for the number of such specifications, either overall or within subsets, and multiplicity-adjusted p-values for each individual specification (strong family-wise error rate control). In this way, researchers can explore multiple reasonable specifications and draw selective conclusions without inflating error rates.

 

Jelle Goeman
Cluster extent inference revisited: quantification and localisation of brain activity

With: Paweł Górecki, Ramin Monajemi, Xu Chen, Thomas E Nichols, Wouter Weeda

Cluster inference based on spatial extent thresholding is a popular analysis method multiple testing in spatial data, and is frequently used for finding activated brain areas in neuroimaging. However, the method has several well-known issues. While powerful for finding regions with some activation, the method as currently defined does not allow any further quantification or localisation of signal. In this paper, we repair this gap. We show that cluster-extent inference can be used (1) to infer the presence of signal in any region of interest and (2) to quantify the percentage of activation in such regions. These additional inferences come for free, i.e. they do not require any further adjustment of the alpha-level of tests, while retaining full family-wise error control. We achieve this extension of the possibilities of cluster inference by embedding the method into a closed testing procedure, and solving the graph-theoretic k-separator problem that results from this embedding. We demonstrate the usefulness of the improved method in a large-scale application to neuroimaging data from the Neurovault database.