Leaderboard comparability after the June 8 real-time data-access fix — were pre-fix scores rescored?

Competition ADIA Lab
1

Hi CrunchDAO team,

I’d like to ask for clarification on how the leaderboard treats submissions evaluated before the real-time data-access issue was fixed on June 8.

Background, as I understand it:

This challenge’s Real-Time Edition is defined by its streaming protocol: infer() must emit a score for each online observation as it arrives, using only the history and the observations seen so far. Before the fix (discussed in 2026-W23 - Structural Break fixes ), it was possible for a submission to access the full online series in advance during inference — i.e., values at timesteps that had not yet “arrived” under the intended protocol.

Why this matters for comparability:

Under the time-stratified AUC metric, lookahead access is not a minor advantage — a detector that can see past the current timestep t can identify a break retroactively (e.g., from the post-break segment’s full statistics, or even from the location of distributional change anywhere in the series) rather than detect it in real time from a short, noisy post-break window. Scores produced under those two regimes measure fundamentally different tasks, and differences of several points of TS-AUC can plausibly come from the access difference alone rather than from modeling quality. Notably, none of the public post-fix baselines and write-ups I’m aware of exceed ~0.61, while the top of the leaderboard stands meaningfully higher — which may be entirely legitimate, but is hard to interpret without knowing which evaluation regime produced which score.

My questions:

  1. Were submissions evaluated before June 8 invalidated, rescored under the corrected harness, or left on the leaderboard as originally scored?
  2. If left as-is, could you add (or point to) a way to distinguish pre-fix from post-fix evaluations — a submission-date column, a badge, or a filtered leaderboard view?
  3. Will final rankings and prizes be computed exclusively from evaluations under the corrected real-time protocol (including any re-evaluation on fresh data)?
  4. Related, for completeness: the rules discussion in Structural Break Real-Time: may infer use prior completed-series state? clarified what state may be carried across series at inference. Is it correct that the determinism re-check (re-running a random subset of series) is the enforcement mechanism for both that rule and the streaming protocol generally?

I’m asking purely for calibration purposes — it’s difficult to judge where genuine methodological headroom lies without knowing whether the current top scores are comparable to post-fix submissions.

Thanks for the great competition!

2

Hello Ali,

  1. All predictions were invalidated.
    The reason is that the leak happened during the infer() process, so changing the scoring mechanism would not have made a difference.
  2. The leaderboard doesn’t consider them at all. So there’s no need to identify them.
  3. Yes.
    The runner will behave the same way for the Out-of-Sample as it does now (already fixed).
  4. I am not sure to properly understand your question I apologize. Please read below.

There seems to be some confusion, so let me explain a bit more:

The leaked lengths

One of the participants reported a method for determining the length of the time series currently being processed. He was suspicious that a team had somehow gained an advantage, and he was right.

After further verification, we were indeed able to confirm and fix the vulnerability.

The fix changed how data points were delivered. It now uses a socket, exchanging data with another process. This made all models 3 minute slower overnight, but it protects us against all memory exploration-related attacks.

To fix the 3-minute slowdown, we introduced parallelism support, which made all your models much faster.

The sharing of state

In Structural Break Real-Time: may infer use prior completed-series state?, I confirmed that you can technically persist a state between time series by accumulating values for the next loop. You can do this by doing something similar to:

def infer(...):
    yield

    summaries = []

    for x_historical, x_online in datasets:
        for point in x_online:
            yield result

        summaries.push(...)  # previous timeseries just ended, record a summary

However, it isn’t a good idea because both parallelism support and the determinism check will work against you:

  • The parallelism mechanism splits the 10,000 time series into N workers.
  • The determinism check is only rerun on 10% of the datasets or 1,000 timeseries, once again split into N workers.

Depending on how many workers you have, you risk having different initial values, so relying on them will be inconsistent.

(see visual representation here: Structural Break Real-Time: may infer use prior completed-series state? - #4 by enzo)

Determinism check

It’s a simple rerun of just 10% of the datasets using the same method as before. Your model shouldn’t even know that it’s been run twice.

Then, we collect and compare both predictions. If they do not match with 1e-8 precision, your model is considered non-deterministic and will not be eligible for a reward.

I suggest not saving values to disk and trying to detect them on the next launch, as we will disqualify those kinds of mechanisms.

I hope I answered all your questions properly, but if you still have some, please continue this thread.

1 Like
Structural Break Real-Time: may infer use prior completed-series state?