Where does the remaining headroom live? Sharing measured ceilings from a whitening-based approach

[weak-van-weyenbergh]

Hi all,

I’m currently around 0.6263 on the public leaderboard with a whitening-based approach, and I’ve spent the last few weeks measuring ceilings rather than chasing features. I’m sharing the numbers here because they closed a lot of directions for me, and I’m hoping someone can tell me where my reasoning breaks.

Setup. Rosenblatt whitening fitted on the historical segment (AR of order ≤ 12 selected by BIC, EWMA scale, empirical CDF of the innovations), roughly 119 window features on the whitened stream, and a blend of a shallow GBM, small MLPs and extremely randomised trees.

Measured ceilings. All of these are out-of-sample, using the real TS-AUC on the real inference path, with the production training recipe:

• An oracle that knows τ and reads five standardised moments of the post-break window beats my model by only +0.019.
• The same oracle integrating over τ instead of knowing it beats me by +0.0035. So roughly 82% of any oracle’s advantage is the structural price of not knowing τ — and that ratio held across several different statistics I tried, which surprised me.
• Running the same statistics on the raw series instead of the whitened one is about 2.5× worse, and the gap widens at longer ages.
• The length of the online segment appears unpredictable from the historical segment (negative out-of-fold R² across 23 descriptors), so no usable prior on P(τ ≤ t).
• Model capacity and model class: at most +0.0007, across five independent probes (GRU, dilated causal TCN, causal transformer, column bagging, large ensembles).

One thing that caught me out and might be useful to others: a more general statistic is not always a stronger oracle. A full empirical-law statistic on cells was noticeably worse than five standardised moments at long ages, purely because of small-sample noise — and long ages carry most of the metric weight.

My question. The top scores are around 0.652, which is about 0.006 above my τ-knowing oracle. So either there’s a class of information none of these ceilings covers — time-series foundation models, identifying the generating family and using a directed rather than omnibus test, something else — or I’m constructing these oracles wrong somewhere.

If anyone has a view on which it is, I’d genuinely like to know. Happy to share more detail on any of the measurements.

Also: will solutions be published after the deadline, as they were for the 2025 edition?

Thanks.

[enzo]

Hello Van Weyenbergh,

You should also share your post in the Discord!

Regarding sharing solution, it will be entirely based on if winners want to share a write up of their work, but it’s not mandatory.

[agreed-harshveer]

Hi Van Weyenbergh,

Three of your four ceilings replicate here by an independent route, on the real-time
edition. I’ve been doing the same thing — measuring what can’t work rather than trying
things — and I have one ceiling you don’t, plus a phenomenon I haven’t seen described
anywhere that I think explains why both of us keep getting zeros.

Setup, so the numbers are comparable. 1,728 features, LightGBM, 12-config ensemble,
61.95 public. Everything below is n=10,000 out-of-fold, 5-fold, paired bootstrap over
series. Paired resolution is ±0.19; absolute resolution is ±0.95, which is why I only quote
paired differences.

1. Six measured defects, six near-zero returns.

This is the part I’d most like a second opinion on. Each defect was located, quantified, and
its worth predicted before a fix was built. Every fix returned approximately nothing.

defect	how it was measured	predicted	actual
null-estimation noise	split-half: 20–23% of large-window feature variance at w≥320 is estimation error	+1.4	+0.016
” (second, independent fix)	same defect, Newey–West shrinkage instead of a simulated universal null	+1.4	−0.20 (cloud)
reference-frame drift	between-series sd 0.332 vs within-series 0.116; benign history→online drift of 0.307 exceeds the median break of 0.194	+0.75	+0.010
window–elapsed mismatch	optimal window marches 5→20→40→160 with elapsed, which is unobservable at inference	+2.8	absorbed
window-ladder hole	a single w=5 feature scores 53.44 at elapsed 1–5, beating the whole 1,728-feature bank’s 51.30	+2.14 in bucket	+0.06
the model’s own errors	a recurrent net given the GBM score + its running max + deviation from running min, trained to correct it	+0.4	+0.095

Four fail by absorption, two by weight share. The mechanism for the first four is that the
bank is enormously redundant:

eight features — one channel, one stream, eight window scales — reach 58.77 out-of-fold.
The full 1,728-feature bank reaches 61.51. The marginal value of the other 1,720 features
is +2.74, and the breadth buying it accounts for 31% of the model’s gain across elapsed
(+10.64 for the single channel against +15.35 for the bank). Same 5-fold protocol, same
series, same rows/rounds/leaves — I had an earlier version of this comparison that trained
the small arm on 1,300 series against a bank trained on 8,000, which entangled breadth with
training size and overstated the gap by 3.7 points.

So the span is already wide enough that a single corrected statistic very likely lies inside
it. That predicts the four zeros rather than explaining them afterwards.

The other two are different: the fix worked and landed where the metric wasn’t looking. The
ladder hole is real and repairable — but elapsed 1–5 is 2.9% of the weight, so the repair is
worth six hundredths of a point.

2. Your τ-oracle result, with a caveat on mine.

Mine came out at −1.33: an oracle handed the exact post-break window length, scoring six
moment statistics with segment length matched between positives and negatives, lost to the
model. But that comparison is confounded — six statistics with known τ against 1,728 features
without it conflates τ-knowledge with statistic richness. Your version is the cleaner test and
reaches the same conclusion, which I think strengthens both. Change-point localisation is
worth nothing here.

Related, and possibly useful: the optimal window tracks elapsed (5 → 20 → 40 → 160), and
elapsed is unobservable. But an oracle allowed to pick the elapsed-matched window still loses
to the full bank by 1.97 weighted. The bank routes around the mismatch.

3. A measurement artifact worth knowing if you’re fitting blend weights.

I caught myself choosing a scalar blend weight on the same data I was scoring it on, and it
cost 0.072. Same candidate: +0.215, CI [+0.007, +0.359], both halves positive, when the
weight was picked globally; +0.143, CI [−0.028, +0.344] once the weight was fitted only
on the half it wasn’t scored on. A t-adaptive version with four free parameters was worse
still (+0.067). I retracted the candidate.

Earlier I had one clear at +0.203 with CI [+0.007, +0.398] — the only thing in the project
that ever excluded zero — and I killed it after it flipped sign on disjoint halves
(+0.142 / −0.167). I’d previously “replicated” it at +0.092 by re-running with fresh folds
over the same series, which is worthless: the two estimates share their sampling noise.
Only disjoint samples test replication. Also worth noting the multiplicity — across ~30
candidates at SE≈0.10, the maximum under a global null sits near +0.20, which is exactly
where that one landed.

4. A fifth ceiling: the bank is data-limited, and its limit is below the top.

Training on random subsets, replicated across three independent draws (noise floor ±0.45):

  1,000 series -> 59.06     5,000 -> 62.52
  2,500 series -> 61.49     7,500 -> 63.20

Regressing on 1/√n gives AUC ≈ 65.6 − 206/√n, predicting the interior points to within
0.03 and 0.14. On a split that reads ~2.2 optimistic against proper cross-validation, that
puts this feature bank’s infinite-data ceiling around 64 in public-leaderboard terms —
below the current top. Not “we need more data”: with unlimited series this representation
still doesn’t get there.

That’s evidence for your closing reading rather than against it. The gap is a class of
information none of these ceilings cover.

5. The question.

You get 0.6263 with 119 features. I get 0.6195 with 1,728, and — importantly — on
substantially the same representation: my bank runs every channel over six parallel streams,
three AR residual orders (2/5/12), a GARCH(1,1)-standardised stream, a normal-score stream
(empirical CDF of the innovations), and raw. Per-series BIC selection of the AR order I tested
directly: −0.002. So the whitening isn’t the difference.

And it isn’t that my good features are buried: cutting my bank by gain importance hurts
monotonically — top-384 −0.09, top-192 −0.55, top-64 −1.34, top-32 −1.84. Every subset is
worse than the whole. So your 119 are not a subset of my 1,728; they’re a different
construction.

So: are your 119 per-(channel × stream × window) cells like mine — a statistic in a window,
quantile-scored against a historical null — or are they aggregated across scales? 119 is
roughly what you’d get by aggregating along an axis instead of enumerating it.

One sentence on that settles in a day what a week of guessing won’t.

Last thing: your +0.0007 across five capacity/model-class probes stopped me pushing a
16-model ensemble I had staged and ready — same family, and my own local estimate for it was
+0.03, already inside noise. It also brackets a diagonal state-space model I’d measured at
+0.143 and couldn’t distinguish from zero. Two dead ends closed by your post before I spent
the submissions, so thanks for writing it up.

Happy to share any of the above in more detail — the absorption table especially, since I
suspect it generalises past this competition.

[agreed-harshveer]

@agreed-harshveer

So: are your 119 per-(channel × stream × window) cells like mine — a statistic in a window,
quantile-scored against a historical null — or are they aggregated across scales? 119 is
roughly what you’d get by aggregating along an axis instead of enumerating it.

Follow-up to my own question, because a measurement I ran since changes it.

First, a correction: I quoted a cross-channel share of 55%. That figure had training size entangled with feature breadth — the narrow arm trained on 1,300 series where the full bank had 8,000. Matched properly it’s 31%.

The number that came out of the matched run is the one that matters here: eight features reach 58.77 where my 1,728 reach 61.51. 216× the features for 2.74 AUC.

So my earlier question was wrong. Whether your 119 are cells or aggregates doesn’t matter much, because on this problem the marginal return to breadth is nearly flat past the first handful. The competitive spread has to live in the estimator, not the enumeration. Your own result points the same way — five standardised moments beating a full empirical-law statistic at long ages is the same phenomenon from the other side.

So the question is just: what do your individual features measure? Not how many, not how they’re arranged.

[weak-van-weyenbergh]

Thanks — this is more useful than my own last three weeks. Answering the
direct question first, then two places where your results replicate on my
side.

What my features actually measure

Your guess is right: they’re per-(channel × window), but aggregated over
the window axis rather than enumerated. 119 columns, nine blocks, all on a
Rosenblatt-whitened stream z — AR(p≤12) by BIC + EWMA conditional scale +
empirical CDF of standardised innovations, everything fitted on the
break-free history only:

block	n	what it measures
rosen12	20	prefix moments + max over 7 dyadic scales of
bayesv12	12	Bayes factors integrated over τ (16 window lengths) and over break amplitude; 3 channels (mean, scale, linear dependence)
rosenq	20	same as rosen12 on a second whitening with heavier tails
rosen4	21	long lags, tail counts beyond 2.5σ/3σ, goodness-of-fit state
rosenb	8	two-directional scale sweep, closed-form likelihood
rosent	8	trajectory of the scale channel
histx	16	history descriptors × current scale statistic
rosenl	9	full empirical law (CvM / AD / KS, 15 thresholds), max over scales
calib	5	per-series null variance of the scale channel

So: max, or Bayesian integration over τ, along the window axis. Never
enumeration. Which — given your own absorption result — is the axis that
turns out not to matter.

Your absorption result replicates here, three ways

You measured 8 → 58.77 and 1728 → 61.51: +2.74 for 216× the columns.

• learning curve in columns: +0.0003 per doubling. 40 → 89 is
  +0.0016; 60 → 89 is +0.0002, and the curve actually peaks at 60.
• best-10 of my 89 reaches 0.6264 against 0.6289 for all 89. A dozen columns
  carry ~97% of it.
• the sharpest one: I built 132 columns from a completely different recipe
  (six transforms × two base series × eight statistics, no whitening in
  common with my pipeline). Alone they reach 0.6199 against my 0.6289.
  Added to my 89 → +0.0007. Different construction, same content.

Closing a 0.026 gap at +0.0003/doubling would need 87 doublings. The column
axis is closed, and we closed it independently.

Your fifth ceiling replicates — and lands on the same number

Learning curve in series: train on subsets, score on 1973 series never
touched by anything, three seeds per point.

n = 1000 0.5889 ± 0.0041
n = 2000 0.5953 ± 0.0053
n = 4000 0.6127 ± 0.0072
n = 7917 0.6171 ± 0.0010

fit AUC = 0.6334 − 1.477/√n residual RMS 0.0031 (within seed noise)

Ceiling at infinite series: 0.6334 in those units (89 columns, virgin
control). The same protocol with the full 119 columns reads +0.0071 higher,
and my virgin-control → leaderboard offset is +0.0029 (0.6242 vs 0.6271 on
the same model). That puts my representation’s infinite-data ceiling at
~0.643 on the leaderboard.

Yours: ~0.64. Mine: ~0.643. Current top: 0.6524.

Two representations with almost nothing in common — 1728 enumerated cells vs
119 aggregated ones — extrapolate to the same ceiling, ~0.01 below the
leader. Caveat I’ll state myself: it rests on the 1/√n form holding to
infinity, fitted on four points. That’s the weakest link in both arguments.

Three results that might save someone time

A directed test buys exactly zero. I gave an oracle τ and the channel
the break occurred in (measured at the oracle on the true pre/post windows),
against the same oracle summing all five channels blind:

omnibus +0.0191
directed +0.0191 difference +0.0000

Break-type distribution, for reference: dependence 54%, volatility 24%,
tails 11%, mean 8%, variance 3%.

The reason is structural to the metric: cross-sectional AUC at fixed t ranks
a positive against negatives, which have no type. Knowing the type sharpens
the numerator of a test you never run. So break-family classification,
per-series test selection and mixture-of-experts routing are closed here — I
handed the oracle the answer for free and it gained nothing.

Integrating over unknown τ costs ~82% of any oracle’s margin. Measured on
two unrelated statistics: 80% on a cell-based empirical law, 82% on five
standardised moments. Same constant. Whatever you build with τ known, budget
a factor of five when you integrate it out.

Rebuilding an oracle’s exact statistic as a feature converts at ~10%.
Two blocks, both correct (H0 calibration verified, mixture distinct from the
max): +0.0004 and +0.0003 against +0.0035 available at the oracle — and in
both cases the gain was negative in the age bands the oracle was winning.
The evidence was already in the model under a different parameterisation.

On blend weights

Your 0.072 loss from fitting weights on the scoring data has a twin here. I
found +0.0165 from a weight search and refused to believe it: the model I was
scoring with had been trained on 9890 series, and my “virgin” 1973 control
was inside that training set. The whole gain was contamination. Retraining on
7917 only dissolved it to zero.

Worth stating generally: check your holdout’s provenance against the training
set of the specific artefact you’re scoring with, not against your CV split.

[weak-van-weyenbergh]

Correction to my own instrument first, because it’s the transferable part: my
series curve had been run with n_estimators=4000, colsample_bytree=0.7 — a
stale config that overfits at small n — and on the tree alone rather than the
production stack. At n=1000 the stale curve read 0.5889; the real one reads
0.6100. The distortion lands exactly where a learning-curve fit is most
leveraged: the 1/√n coefficient drops from 1.477 to 0.670.

Redone at the production configuration: 6 sizes, 4 seeds, full stack, 119
columns, scored on 1973 series never touched by anything.

600 0.6051 ± 0.0073 2800 0.6190 ± 0.0052
1000 0.6100 ± 0.0069 4700 0.6233 ± 0.0029
1700 0.6173 ± 0.0019 7917 0.6239 ± 0.0009

constrained c = 0.50 ceiling 0.6322 b = 0.670
free c = 0.575 ceiling 0.6302 b = 1.013 c in [0.24, 0.95]

Three independent instruments agree at n=7917 (0.6239 / 0.6242 / 0.6240), so
the level is solid; the exponent is what’s uncertain, and it comes out above
0.5. Fitting the first five points and predicting the sixth undershoots by
−0.0013 (−0.0015 on the tree-alone curve). The curve decays faster than
1/√n, so a 1/√n ceiling is an overestimate.

Extrapolated, plus my virgin→leaderboard offset of +0.0026:

×10 series (79k) ~0.632 on the leaderboard
×100 series (790k) ~0.633

A hundred times the data buys this representation about +0.006, and lands at
~0.633 against a leader at 0.6524. My earlier figure of 0.643 was wrong: it
attributed a +0.0071 gap to columns when that gap mixed columns, model class
and configuration.

Two consequences worth stating plainly:

• the data axis is closed here too. My last real segment is +0.0008 per
  doubling of series, against +0.0003 per doubling of columns. A factor of
  three, not the factor of fifteen I had on file — that number came from the
  stale config as well, and it had been steering a synthetic-augmentation
  programme that returned +0.0013 on the stack, exactly what +0.0008/doubling
  predicts.
• if your 206/√n fit behaves the same way, both ceilings move down together.
  The leave-one-out test takes minutes and it’s the one number that decides
  whether “the leader is 0.01 outside both representations” or “0.02”.