DataCrunch 2: are live predictions still scored after switching the selected run?

Hi,

How I’m using the terms:

  • Submission: the code + model files (resources/) I push with crunch push.
  • Models: what’s inside a submission (trained weights, blend weights, the train() / infer() code).
  • Run: one execution of a submission on the platform. The selected run is the one the project is scored with.

As I understand it, the models in a submission can’t be edited in place. To change them I push an updated submission, create a run of it, and select that run. That new run first re-predicts the ~300-moon backfill (unscored), then predicts one live moon per week, and each live prediction is scored ~5 weeks later (week 1 → week 6).

Example: run A (from submission version 1) is selected and predicts live moons for weeks 1–3. In week 4 I change the models, push version 2, and select its run B.

  1. Old run’s pending predictions: Are run A’s live predictions for weeks 1–3 still scored and shown on the leaderboard when they resolve (weeks 6–8), even though A is no longer selected?
  2. Only the selected run? Or is only the currently selected run scored, so the project shows no score until run B’s first live moon resolves (week 9)?
  3. Rewards: Is a week’s payout based on the run that made that week’s prediction, or on the run selected when the score is published?
  4. Switching back: If I select run A again later (it kept running), are its pending predictions scored again, or does it start over?
  5. Changing models without a new run: Is there a way to update the models of the submission behind the selected run (e.g. replace files in resources/) without creating a new run and redoing the backfill?

Background: I want to know how often I can change the models without leaving weeks with no score.

Thanks!

Hello Bin,

You understood pretty much everything, let me answer your questions:

  1. That is correct. Even though the model isn’t selected anymore in the present, we are resolving and showing targets in the past. Meaning your “old” selection still applies.

  2. (answered in 1.)

  3. Since it is a leaderboard in the past, it is based on the Run that was selected in the past.

  4. Yes, that does not change anything.

  5. No, and that is by design.

  6. “Background”: Basically every week. By default, if you do not wish to update anything, we will always take the previous week’s run as a continuation, basically like a chain.
    One thing to note: if you reselect an old Run that already passed the historical ~300 moons, they will not have to re-infer them.

Thanks enzo for the swift reply. That answers all my questions.