For the current Structural Break Real-Time competition, may `infer()` maintain a **forward-only summary of series that have already been completely processed** and use that summary when scoring later series?
The proposed state would:
- run with parallelism 1;
- contain only clean-history summaries and prediction trajectories from previously completed test series;
- never use test labels, data files, a future point of the current series, or the current series’ final horizon;
- yield each prediction before consuming the next online point; and
- never revise any earlier prediction.
Or must every series’ predictions be independent of all other test series?
If forward-only cross-series state is allowed, is the series order guaranteed to be stable/deterministic in public and out-of-sample scoring?
Yes, that’s totally doable. You can detect when a time series is over after the for loop:
def infer(...):
yield
for x_historical, x_online in datasets:
for point in x_online:
yield result
... # previous timeseries just ended, record a summary
Even though the order is deterministic, making it easier for participants to debug their code, we never mentioned one. Therefore, your code should function the same way whether the order is the same or random.
For the Out-of-Sample, your model will only be run once on a brand new dataset. I’m not sure if how a stable ordering would be useful in that case.
Following up on the earlier answer that infer() may keep a forward-only summary of already-completed test series and use it when scoring later ones: how does that interact with the determinism requirement (re-run on 10% of the data must reproduce predictions within 1e-8, and non-deterministic output is ineligible for rewards)?
A prediction that uses a forward-only summary of previously completed series is deterministic given the dataset, but a 10% subset contains a different set of prior series, so predictions on the common series would differ by more than 1e-8 between the full run and the 10% re-run. Is that acceptable (i.e., the check compares each run against itself / a re-run on the same subset), or does the 10% re-run need to reproduce the full-run predictions — which would effectively rule out any use of cross-series state despite it being mechanically allowed?
