# Leaderboard comparability after the June 8 real-time data-access fix — were pre-fix scores rescored?
**URL:** https://forum.crunchdao.com/t/leaderboard-comparability-after-the-june-8-real-time-data-access-fix-were-pre-fix-scores-rescored/1188
**Category:** ADIA Lab
**Created:** [August 8, 2026, 7:11pm UTC](https://forum.crunchdao.com/t/leaderboard-comparability-after-the-june-8-real-time-data-access-fix-were-pre-fix-scores-rescored/1188 "2026-08-08T19:11:00Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@kamer-ali](https://forum.crunchdao.com/u/kamer-ali)
#### Post date: [August 8, 2026, 7:11pm UTC](https://forum.crunchdao.com/t/leaderboard-comparability-after-the-june-8-real-time-data-access-fix-were-pre-fix-scores-rescored/1188/1 "2026-08-08T19:11:00Z")
Hi CrunchDAO team,
I’d like to ask for clarification on how the leaderboard treats submissions evaluated before the real-time data-access issue was fixed on June 8.
Background, as I understand it:
This challenge’s Real-Time Edition is defined by its streaming protocol: infer() must emit a score for each online observation as it arrives, using only the history and the observations seen so far. Before the fix (discussed in [2026-W23 - Structural Break fixes](http://forum.crunchdao.com/t/1156) ), it was possible for a submission to access the full online series in advance during inference — i.e., values at timesteps that had not yet “arrived” under the intended protocol.
Why this matters for comparability:
Under the time-stratified AUC metric, lookahead access is not a minor advantage — a detector that can see past the current timestep t can identify a break retroactively (e.g., from the post-break segment’s full statistics, or even from the location of distributional change anywhere in the series) rather than detect it in real time from a short, noisy post-break window. Scores produced under those two regimes measure fundamentally different tasks, and differences of several points of TS-AUC can plausibly come from the access difference alone rather than from modeling quality. Notably, none of the public post-fix baselines and write-ups I’m aware of exceed ~0.61, while the top of the leaderboard stands meaningfully higher — which may be entirely legitimate, but is hard to interpret without knowing which evaluation regime produced which score.
My questions:
1. Were submissions evaluated before June 8 invalidated, rescored under the corrected harness, or left on the leaderboard as originally scored?
2. If left as-is, could you add (or point to) a way to distinguish pre-fix from post-fix evaluations — a submission-date column, a badge, or a filtered leaderboard view?
3. Will final rankings and prizes be computed exclusively from evaluations under the corrected real-time protocol (including any re-evaluation on fresh data)?
4. Related, for completeness: the rules discussion in [Structural Break Real-Time: may infer use prior completed-series state?](http://forum.crunchdao.com/t/1186) clarified what state may be carried across series at inference. Is it correct that the determinism re-check (re-running a random subset of series) is the enforcement mechanism for both that rule and the streaming protocol generally?
I’m asking purely for calibration purposes — it’s difficult to judge where genuine methodological headroom lies without knowing whether the current top scores are comparable to post-fix submissions.
Thanks for the great competition!
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [August 8, 2026, 8:11pm UTC](https://forum.crunchdao.com/t/leaderboard-comparability-after-the-june-8-real-time-data-access-fix-were-pre-fix-scores-rescored/1188/2 "2026-08-08T20:11:53Z")
Hello Ali,
1. All predictions were invalidated.
The reason is that the leak happened during the infer() process, so changing the scoring mechanism would not have made a difference.
2. The leaderboard doesn’t consider them at all. So there’s no need to identify them.
3. Yes.
The runner will behave the same way for the Out-of-Sample as it does now (already fixed).
4. I am not sure to properly understand your question I apologize. Please read below.
* * *
There seems to be some confusion, so let me explain a bit more:
## The leaked lengths
One of the participants reported a method for determining the length of the time series currently being processed. He was suspicious that a team had somehow gained an advantage, and he was right.
After further verification, we were indeed able to confirm and [fix the vulnerability](https://github.com/crunchdao/competitions/pull/79/changes).
The fix changed how data points were delivered. It now uses a socket, exchanging data with another process. This made all models 3 minute slower overnight, but it protects us against all memory exploration-related attacks.
To fix the 3-minute slowdown, we introduced parallelism support, which made all your models much faster.
## The sharing of state
In [Structural Break Real-Time: may infer use prior completed-series state?](http://forum.crunchdao.com/t/1186), I confirmed that you can technically persist a state between time series by accumulating values for the next loop. You can do this by doing something similar to:
```auto
def infer(...):
yield
summaries = []
for x_historical, x_online in datasets:
for point in x_online:
yield result
summaries.push(...) # previous timeseries just ended, record a summary
```
However, it isn’t a good idea because both parallelism support and the determinism check will work against you:
- The parallelism mechanism splits the 10,000 time series into N workers.
- The determinism check is only rerun on 10% of the datasets or 1,000 timeseries, once again split into N workers.
Depending on how many workers you have, you risk having different initial values, so relying on them will be inconsistent.
(see visual representation here: [Structural Break Real-Time: may infer use prior completed-series state? - #4 by enzo](https://forum.crunchdao.com/t/structural-break-real-time-may-infer-use-prior-completed-series-state/1186/4))
## Determinism check
It’s a simple rerun of just 10% of the datasets using the same method as before. Your model shouldn’t even know that it’s been run twice.
Then, we collect and compare both predictions. If they [do not match with 1e-8 precision](https://github.com/crunchdao/competitions/blob/a92aee2a7e9f4943517a6277f45a4c7e5d9bc46b/competitions/structural-break-real-time/scoring/runner.py#L146-L157), your model is considered non-deterministic and will not be eligible for a reward.
I suggest not saving values to disk and trying to detect them on the next launch, as we will disqualify those kinds of mechanisms.
* * *
I hope I answered all your questions properly, but if you still have some, please continue this thread.