Question: Is the full cloud X_test distribution representative of the reduced test sample?

Competition ADIA Lab
1

I ran into something during a local data audit that I think is worth clarifying with the competition team.
In the training data, there is a very small extreme-severity tail: roughly 0.5% of the sequences have extremely large changes in online variance, with some online standard deviations reaching ~2,000. However, in the reduced test sample distributed with the competition, we see none of these extreme-variance sequences. Its maximum online standard deviation is only around 1.95. So there appears to be a noticeable difference in the observed severity distribution between TRAIN and the reduced TEST sample.

The cloud evaluation uses the full `X_test`, while participants only have access to a reduced test sample locally. This creates an uncertainty that we currently cannot resolve:

Is the full `X_test` used by the cloud evaluator broadly representative of the reduced test sample, or does the full test set contain regimes/severity levels that are not present in the reduced sample? This is particularly relevant when interpreting local experiments. We don’t want to assume that a local improvement (or failure) will translate to the cloud if the local test distribution is materially different from the actual evaluation distribution.

Does the competition team provide, or could it provide, any 'non-sensitive aggregate statistics’describing the full `X_test` distribution? We don’t need the raw test data or labels. Even something as simple as aggregate statistics for:

- `T`

- `hist_len`

- `online_std`

- `logvar`

  • `logmad`

- `logrange`

- `maxabs`

would probably be enough. For example:

`N / mean / std / p01 / p05 / p10 / p25 / p50 / p75 / p90 / p95 / p99 / max`

would allow us to compare:

`TRAIN vs reduced TEST vs full cloud TEST`

without exposing any test observations or labels.

One important clarification

I’m ‘not asking for test labels, tau/breakpoint information, raw sequences, or anything that would allow us to reconstruct the hidden test set’.

The goal is simply to understand whether the locally available reduced test sample is representative of the distribution on which the leaderboard is actually evaluated.

If these statistics cannot be provided, it would also be useful to know whether there is another officially supported way for participants to assess this distributional comparability. Thanks!

2

Hello Dongling,

The overview contains all the information we can share, such as the average length (from 10 to 1,000 data points) and the fact that there are a total of 10,000 unique timeseries.

Statistics cannot be shared for a simple reason: in a real environment, these time series are infinite. They are only limited for the purposes of the competition.

There is no official method by which participants can extract this data. Even if you could, the Out-of-Sample would only provide a single pass. It would be better to do without them.

We were not provided with any information on how the datasets were built. The timeseries given in the reduced set may or may not match those in the training set. (It’s not a hidden clue, we really don’t know!)

In all cases, you should avoid relying on the reduced X_test set. Its only purpose is to check your code locally; it should not be used for (even if very tiny) overfitting.