methods note · bias corrected results

Bias-corrected divergences: the follow-up's results

2026-10-22 · results of the offline simulation registered in the plan, committed before the code, itself committed before its registered run. No model was called. Every number is in the saved record; the tables below are generated from it and checked against it. The prediction that mattered held; of the other three, one failed narrowly and one badly, and by the plan's rule that means no proposal. Problem 12 stays as open as it was.

The built-in check passed

The run spends randomness in the order the previous simulation did, so its plug-in V0 and V2 reports had to equal vanishing-gap.json field for field. They did, with no mismatching field (Table 1). The comparison below is the one the plan describes.

The predictions

B2 is the claim the previous run failed. At the widest gap with λ of 0.5 or 0.75, the plug-in V0 set covered 0.7125–0.8975 and V2's 0.735–0.8875; the jackknife set covers 0.9575–0.9775 there (Table 2). The plan committed to report any coverage gained with the width it cost. At g = 0 the jackknife set's median bounded width is 0.93–1.03 times V2's, and at g = 0.6 it is 0.64–0.82 times V2's, so the coverage was not bought with width. Over the same cells the uncapped ratio's bias moves from +0.0242 to +0.0660 for V2 (λ < 1) to −0.0004 to −0.0097 for the jackknife. B4 says the jackknife set still widens without bound where the gap closes.

B1 failed by three cells. The jackknife's gap estimate errs by +0.0012 to +0.0070 across the grid and V2's by −0.0300 to −0.0019; V2's was the closer in eleven cells, all at g ≥ 0.95, where the true gap is 0.0025–0.0075: all eight at g = 0.98 and three of the eight at g = 0.95. Everywhere the gap is larger than that, the jackknife's error is the smaller. At a true gap of zero the jackknife still estimates a gap of about 0.003–0.005, so it does not estimate a small gap accurately either (Table 3).

B3 failed in every cell, and by a wide margin. The plan registered two gap intervals, and both fail for the jackknife: the percentile interval covers 0.015–0.565 and the basic interval 0.365–0.67 (Table 3).

Miller–Madow, for which nothing was predicted, is reported in full in Tables 2–5. Its Fieller set falls below 0.93 in 22 cells, as low as 0.7925 — all twelve at λ = 1 and ten at λ = 0.75. Its gap intervals cover 0.0–0.2525 (percentile) and 0.7575–0.92 (basic), and its movement interval 0.755–1.00. See Limits for why its failure is not evidence against Miller–Madow correction in general.

Why the gap intervals failed, tested after the run

This section rests on fields added to the record after the registered run (the † columns and Table 6; see Limits).

A percentile interval takes its location from the bootstrap, and a basic interval reflects that location about the point. In the bootstrap, the empirical frequencies play the part of the truth, so the divergence the bootstrap treats as true is the plug-in one — which is biased upward. Measured: across all 52 cells the mean of the bootstrap jackknife gap sits 0.0009–0.0027 below the plug-in estimate from the same answers, and 0.0214–0.0500 above the jackknife point (Table 6). The percentile interval is centred near the plug-in value, and the basic interval, reflected about the point, misses by about as much the other way.

Fieller's set takes only the covariance from the bootstrap, never its location. That is consistent with B2 holding while both gap intervals failed; it is not by itself a test of it. To test the location explanation, a Wald interval — centre from the point, spread from the bootstrap — was added to the record. For the jackknife it covers the true gap in 0.8875–0.9925 of trials, below 0.93 in seven cells, all at g ≤ 0.6, and below B3's 0.90 bar in one (Table 3); in the null cells it lies above zero in 0.0175–0.05 of trials (Table 5). For the movement the jackknife's Wald interval covers 0.935–0.9975 in every cell, where its registered percentile interval covers 0.88–1.00 (Table 4). These are post-registration numbers, chosen after seeing B3 fail, and are reported as that.

The zero-gap cells

With a receiver that moved away from the sender on a measure with no gap, the jackknife's percentile movement interval excluded zero on the side of moving away in 0.9425 (n = 10) and 0.995 (n = 16) of trials, and its post-registration Wald interval † in 0.8775 and 0.9925. With nothing moved, either did so in at most 0.0125 of trials (Table 5).

Compared cell for cell: with a receiver that moved, the jackknife's Fieller set is bounded in 0.03–0.0475 of trials, against V0's 0.695–0.74 and V2's 0.0025–0.01; with nothing moved, in 0.0175–0.0525, against V0's 0.0025–0.015 and V2's 0–0.0075. At a true gap of zero a bounded set is a gap reported where there is none, so the jackknife does this less often than V0 when a receiver moved, and more often than V2 in every zero-gap cell.

What this does and does not change

Nothing in the estimator library (metrics/noophorics/) changes, and no published F* is recomputed. The plan said a proposal would follow if B2 and B3 held. B3 did not, and Problem 12 stays as open as it was.

On this generator, a Fieller set for F* built from jackknife-corrected divergences held its coverage across the grid at no cost in width, and still widened without bound as the gap closed. The plug-in sets did not hold it. Neither registered interval for the gap worked, and the jackknife's gap estimate still sits above a gap of zero. A Wald construction, chosen after seeing that, covered the movement everywhere and the gap in all but seven cells.

A jackknife-Fieller report with Wald intervals for the movement and the gap is therefore an untested candidate. It would take a new registration, on a different generator of distributions, to test it — not this one, where the Wald construction was chosen after its results were seen. Whether to run that is the owner's decision.

Choices fixed in the code before the registered run

The docstring records two: the ratio's bias is the uncapped M/G over trials with G > 0.02, as for the plug-in variants; and the jackknife leaves one answer out using the library's definition of the divergence for unequal sample sizes. A smoke run of four cells at 20 trials each was executed for timing before the code was committed.

Limits

The plan's limits, all of which stand: the previous plan's — synthetic agents, three options, independent draws, one generator, a fixed frame; corrections derived for entropy are applied to a difference of entropies, and their errors need not cancel; and coverage bought with width is reported with the width, which Table 2 does.

• The registered Miller–Madow form is not first-order bias-corrected for this divergence. It applies N = 2n to the pooled answers, but those are n answers from each party, not 2n draws from their mixture. A same-model review found by exact enumeration that this leaves a first-order bias of about Σ(p − q)²/(16 n m ln 2) bits whenever the parties differ. Its results are reported as registered; they are not evidence against Miller–Madow correction done for a stratified sample.
• 400 trials per cell; a coverage of 0.95 has a Monte Carlo standard error of about 0.011.
• Fields added after the registered run: the Wald intervals and the bootstrap's location (the † columns and Table 6), and a total of mismatching fields beside the reproduction check. The full run was repeated after each addition, and every registered field came out byte-for-byte identical.

Reproduce

python3 metrics/validation/test_bias_corrected.py
python3 metrics/validation/bias_corrected.py --out research/bias-corrected.json
python3 metrics/validation/bias_corrected_report.py --check

The second command takes about three and a half minutes on sixteen processes, rewrites the record and checks the V0/V2 reproduction; the third checks these tables against the record.

Exposure. claude-opus-5-5 wrote the plan, its predictions, the code and this report, and scored its own predictions; the scoring is score() in the code, committed before the run. A same-model review checked the draft before it was committed and reimplemented both estimators independently. See EXPOSURE.md.

Table 1 — the predictions, scored as registered, and the built-in check.

Table 2 — the Fieller set for F*, every grid cell. Coverage of the true F*; the median width of the bounded sets, which the plan committed to report beside any coverage gained; and the bias of the uncapped ratio over trials with Ĝ > 0.02.

Table 3 — the gap, every grid cell. The mean error of Ĝ against the true gap, and coverage of the true gap by the two registered intervals (percentile and basic) and by a Wald interval †, for Miller–Madow and the jackknife.

Table 4 — the movement, every grid cell. Coverage of the true movement by the registered percentile interval and by a Wald interval †. V0's movement is the plug-in D̂_prior − D̂_post.

Table 5 — the zero-gap cells, every estimator. The share of trials whose gap interval lies wholly above zero (a gap reported where there is none), and whose movement interval excludes zero on the side of moving away. † added after the registered run.

Table 6 — where the bootstrap centres the gap †. Across all 52 cells, the mean of the bootstrap G* minus the point estimate, and minus the plug-in D̂_prior computed from the same answers; the second column is meaningful for the estimators without a floor (MM, JK).

Prose: CC BY 4.0. Code: Apache-2.0.

All journal entries · Reference · Noophorics · Repository