Leaderboard comparability after the June 8 real-time data-access fix — were pre-fix scores rescored?

Competition ADIA Lab
2

Hello Ali,

  1. All predictions were invalidated.
    The reason is that the leak happened during the infer() process, so changing the scoring mechanism would not have made a difference.
  2. The leaderboard doesn’t consider them at all. So there’s no need to identify them.
  3. Yes.
    The runner will behave the same way for the Out-of-Sample as it does now (already fixed).
  4. I am not sure to properly understand your question I apologize. Please read below.

There seems to be some confusion, so let me explain a bit more:

The leaked lengths

One of the participants reported a method for determining the length of the time series currently being processed. He was suspicious that a team had somehow gained an advantage, and he was right.

After further verification, we were indeed able to confirm and fix the vulnerability.

The fix changed how data points were delivered. It now uses a socket, exchanging data with another process. This made all models 3 minute slower overnight, but it protects us against all memory exploration-related attacks.

To fix the 3-minute slowdown, we introduced parallelism support, which made all your models much faster.

The sharing of state

In Structural Break Real-Time: may infer use prior completed-series state?, I confirmed that you can technically persist a state between time series by accumulating values for the next loop. You can do this by doing something similar to:

def infer(...):
    yield

    summaries = []

    for x_historical, x_online in datasets:
        for point in x_online:
            yield result

        summaries.push(...)  # previous timeseries just ended, record a summary

However, it isn’t a good idea because both parallelism support and the determinism check will work against you:

  • The parallelism mechanism splits the 10,000 time series into N workers.
  • The determinism check is only rerun on 10% of the datasets or 1,000 timeseries, once again split into N workers.

Depending on how many workers you have, you risk having different initial values, so relying on them will be inconsistent.

(see visual representation here: Structural Break Real-Time: may infer use prior completed-series state? - #4 by enzo)

Determinism check

It’s a simple rerun of just 10% of the datasets using the same method as before. Your model shouldn’t even know that it’s been run twice.

Then, we collect and compare both predictions. If they do not match with 1e-8 precision, your model is considered non-deterministic and will not be eligible for a reward.

I suggest not saving values to disk and trying to detect them on the next launch, as we will disqualify those kinds of mechanisms.

I hope I answered all your questions properly, but if you still have some, please continue this thread.

1 Like
Structural Break Real-Time: may infer use prior completed-series state?