Closure tests and calibration
L0/L1 statistics derived — expected χ², coverage, the health identity — and why certificates are ratio-specific.
Chapters 4.1–4.3 built a beautiful machine. This chapter is about refusing to trust it. Before an astronomer points a new telescope at the sky, she points it at a source she already knows — a calibration star — and demands the instrument return the truth. Closure testing is exactly that discipline for a fitting pipeline: plant a known truth, run the full machinery blind, and check that it comes back — the right central values, the right $\chi^2$ statistics, bands with the advertised coverage. We derive the expected statistics at each rung (L0: noiseless data; L1: one noisy realization), including the subtle question of what "$\chi^2 \to 0$" can actually mean for a sampler, put error bars on coverage itself, and end with the two honest limits: why a certificate earned at one parameter count does not transfer to another, and what closure can never certify at all.
The logic of planted truth
Pick a truth vector $\theta^*$ inside the model family, generate pseudo-data from it, and fit with the identical pipeline — same priors, same sampler settings, same $\chi^2$, same termination — that will later run on real data. Because the truth is known, every output becomes checkable: the best-fit $\chi^2$ has a computable expected value, the posterior bands have a measurable coverage against $f(x;\theta^*)$, and any systematic deviation is a pipeline pathology by construction, not a physics surprise — there is no model mismatch to hide behind. The ladder ascends in realism: L0, noiseless data $D = T(\theta^*)$, isolates the optimizer and sampler; L1, one noisy draw $D = T(\theta^*) + \eta$ with $\eta \sim \mathcal N(0, C)$, adds the statistical fluctuations of a real experiment while keeping the truth in-family. Our current certificate at 52 parameters is PPDF-11: L0 $\chi^2/N = 0.0074$, L1 $\chi^2/N = 0.979$ — and this chapter derives what those two numbers had to be for the machinery to pass.
With noiseless in-family data, $\chi^2(\theta) = \big(T(\theta^*) - T(\theta)\big)^{\mathsf T} C^{-1} \big(T(\theta^*) - T(\theta)\big) \ge 0$, with equality at $\theta = \theta^*$: the true minimum is exactly zero. So why did our nested-sampling L0 report a total $\chi^2_{\min} = 0.0074 \times 648 = 4.80$ rather than $10^{-8}$? Because a sampler's "best point" is its luckiest draw, not a polished optimum. Near the minimum the posterior is $\propto e^{-\chi^2/2}$ with $\chi^2 = \Delta^{\mathsf T} A \Delta$, so for a draw from the posterior, $\chi^2 \sim \chi^2_{d_{\text{eff}}}$ — a chi-square with one unit per constrained direction, mean $d_{\text{eff}}$ (§4.3's complexity, $\approx 13.7$ here). The best of $n$ effectively independent draws sits at the $\sim 1/n$ quantile, and the small-$c$ tail of a chi-square is
$$ P\big(\chi^2_{d} \le c\big) \;\approx\; \frac{(c/2)^{d/2}}{\Gamma(d/2 + 1)}\, e^{-c/2} $$
For $d = 13.7$, $c = 4.8$: $P \approx 10^{-2}$ — the observed floor is what the minimum over $\mathcal O(10^2)$ independent-equivalent posterior draws should be. The floor scales like the complexity, gently reduced by sample size: a sampler's L0 lands at $\mathcal O(C)$, orders of magnitude above machine zero, and that is correct behaviour. Gradient descent from the sampler's best point is a different animal — it polishes one point down the quadratic bowl to machine precision: our Hessian-GD refit reached $\chi^2 = 1.9\times10^{-8}$. Both numbers certify the same thing at their own resolution: the likelihood surface has its zero where the truth was planted. The gate is set accordingly: L0 passes on $\chi^2/N \ll 1$ (ours: 0.0074), not on $\chi^2 = 0$. $\blacksquare$
Now $D = T(\theta^*) + \eta$, $\eta \sim \mathcal N(0, C)$. Whiten: $e = C^{-1/2}\eta \sim \mathcal N(0, \mathbb 1_N)$, and write the whitened residual at parameters $\theta$ as $e - \tilde J\,\delta\theta$ with $\tilde J = C^{-1/2} \partial T/\partial\theta$ (linearizing around the truth — legitimate because the fit lands near it, by L0). Minimizing over $\delta\theta$ projects $e$ onto the orthogonal complement of the model's tangent space, $\mathrm{col}(\tilde J)$, of dimension $d_{\text{eff}}$ (the constrained directions; flat directions contribute no tangent motion the data can see):
$$ \chi^2_{\min} = \big\| (\mathbb 1 - P)\, e \big\|^2, \qquad P = \text{projector onto } \mathrm{col}(\tilde J) $$
For a standard normal vector, $\|(\mathbb 1 - P)e\|^2 \sim \chi^2_{N - d_{\text{eff}}}$ exactly. Hence
$$ \mathbb E[\chi^2_{\min}] = N - d_{\text{eff}}, \qquad \operatorname{var}[\chi^2_{\min}] = 2\,(N - d_{\text{eff}}) $$
The fit "spends" one unit of $\chi^2$ per constrained direction fitting noise — the same $d_{\text{eff}}$ that appeared as complexity in §4.3, now counted from the other side. An L1 result far below $N - d_{\text{eff}}$ means overfitting or an inflated covariance; far above means the machinery failed to find the planted truth. $\blacksquare$
Coverage asks a sharper question than $\chi^2$: are the bands honest? Define pointwise band coverage: at each of $n$ probe points $x_j$, check whether the truth $f(x_j; \theta^*)$ lies inside the posterior 68% credible interval for $f(x_j)$; report the fraction $\hat p$. If the machinery is calibrated, each check is (marginally) a Bernoulli trial with $p = 0.68$, so the measured coverage carries its own binomial uncertainty:
$$ \hat p \;=\; p \;\pm\; \sqrt{\frac{p(1-p)}{n}} $$
— a calibration experiment with finite statistics. Vocabulary: calibrated, $\hat p$ consistent with 0.68; conservative, $\hat p$ significantly above (bands wider than advertised — inefficient but safe); anti-conservative, $\hat p$ significantly below (bands lie — the one unforgivable failure). Caveat that matters in practice: neighbouring $x_j$ share parameters, so the $n$ trials are strongly correlated and the effective $n$ is far smaller than the point count — the binomial bar is a lower bound on the real uncertainty. Worked Example 2 runs the numbers. $\blacksquare$
In-family vs out-of-family truth
Everything above assumed the truth is representable: $\theta^*$ exists with $T(\theta^*)$ exact. Plant an out-of-family truth instead — pseudo-data from a different parametrization — and the two questions "does the sampler work?" and "can the model bend far enough?" collapse into one confounded test: an L1 excess could be sampler failure or model rigidity, and you cannot tell which. That is why our MSHT20 closure drills (PPDF-11) plant the truth with the model itself: it isolates the machinery, which is the thing a closure test can actually certify. Out-of-family studies are valuable but answer a different question (parametrization bias — the multi-model program of §4.3's averaging section), and mixing the two produces tests that neither certify the sampler nor measure the bias cleanly.
The same Colibri machinery that passed closure at 13 parameters was re-certified from scratch before the 52-parameter runs — and that was not bureaucracy. What closure certifies is the machinery in a posterior regime, and the regime is set by the information ratio $N/d$: at $648/13 \approx 50$ the posterior is a single compact quasi-Gaussian blob; at $648/52 \approx 12$, with only $\sim 14$ directions constrained (complexity 13.7, §4.3), the posterior is a 14-dimensional shell stretched along $\sim 38$ prior-dominated plateaus. Every hard part of §4.2 scales with that geometry: slice decorrelation lengths ($156 = 3\times52$ steps), live-point coverage of flat directions (the 300$\to$1200 population growth), termination on plateaus, weight concentration. A sampler can be flawless in the first regime and silently biased in the second — so the L0/L1 gates were re-run at 52 params, giving PPDF-11's 0.0074 and 0.979. Standing lab rule: new parameter count, new dataset, or new prior geometry $\Rightarrow$ new certificate. Ratios change; certificates do not transfer.
What closure cannot certify
A closure test plants truths the model can represent — so it is structurally incapable of telling you whether the model can represent the proton. That question — adequacy on real data — belongs to goodness-of-fit: compare the real-data $\chi^2_{\min}$ against the same yardstick $N - d_{\text{eff}}$ that L1 derived, now with nature supplying the truth. Our real-data run answers it: $\chi^2/N = 1.002$ (PPDF-12), against an expectation of $(N - d_{\text{eff}})/N \approx 0.98$ with spread $\pm 0.055$ — adequate. The division of labour is exact: closure certifies the instrument, goodness-of-fit judges the model, and the evidence (§4.3) ranks the models. Three questions, three statistics, never interchangeable: a pipeline that passes closure can still carry a wrong model (the toy's $\chi^2/N \approx 5.6$ says so at $+1453$'s expense), and a perfect real-data $\chi^2$ from an uncertified pipeline proves nothing, because uncalibrated machinery can reach good $\chi^2$ with dishonest bands.
Worked example 1 — scoring our own L1 certificate
Expectation: $\mathbb E[\chi^2_{\min}] = N - d_{\text{eff}} = 648 - 11 = 637$, so $\mathbb E[\chi^2/N] = 637/648 = 0.983$.
Spread: $\sigma(\chi^2_{\min}) = \sqrt{2 \times 637} = 35.7$, so $\sigma(\chi^2/N) = 35.7/648 = 0.055$.
Score the measurement: PPDF-11's L1 gave $\chi^2/N = 0.979$, i.e. $\chi^2_{\min} = 634.4$. Pull: $(634.4 - 637)/35.7 = -0.07\sigma$. As calibrated as a single realization can look.
Two lessons hiding in the arithmetic. First, the test is one draw from a distribution with $\sigma(\chi^2/N) = 0.055$: any single L1 between roughly $0.87$ and $1.09$ would also have passed, so one L1 can catch gross failures but cannot certify percent-level calibration — that is what coverage (many probe points) and repeated-realization L1 batteries add. Second, inverting the logic to measure $d_{\text{eff}}$ from one L1 gives $\hat d = N - 634.4 = 13.6 \pm 35.7$ — uselessly wide. The sharp estimate of constrained directions comes from the complexity $C = 13.7$ of the evidence run (§4.3), whose posterior average beats one noisy realization; the L1's job is only to be consistent with it, and (13.6 vs 11 vs 13.7, all within a fraction of $\sigma$) it is.
Worked example 2 — coverage arithmetic on 120 probe points
Binomial band: for $n = 120$ $x$-points and true coverage $p = 0.68$: $\sigma = \sqrt{0.68 \times 0.32 / 120} = \sqrt{0.001813} = 0.043$. So a calibrated pipeline should return measured coverage in $[63.7\%, 72.3\%]$ at $1\sigma$, $[59.5\%, 76.5\%]$ at $2\sigma$ — if the 120 checks were independent.
Score our spread: across flavours we measured 65% to 100%. The 65% flavour sits $0.7\sigma$ below 0.68 — textbook calibrated. The 100% flavour is formally $(1.00 - 0.68)/0.043 = 7.4\sigma$ high — but here the correlation caveat of Derivation 3 bites: adjacent $x$-points of one flavour's curve are driven by the same few parameters, so the effective number of independent checks is closer to the number of constrained directions feeding that flavour — order 10, not 120. With $n_{\text{eff}} = 10$, $\sigma = \sqrt{0.68 \times 0.32/10} = 0.148$, and 100% is a $2.2\sigma$ fluctuation on the conservative side.
Interpretation, stated the way the result card states it: no flavour is anti-conservative (nothing below the band) — the bands never lie downward; some flavours are conservative — bands wider than 68% where the data barely constrain them, which is the prior (honestly) doing the talking, §4.1's flat-direction regime. That pattern — calibrated where constrained, conservative where not — is precisely the signature of a healthy Bayesian pipeline with uninformative directions, and it is what PPDF-11's certificate records. The follow-up that would sharpen it from qualitative to quantitative: many-realization L1 batteries to measure $n_{\text{eff}}$ per flavour directly — logged as future work in the wiki.