Hi,
How I’m using the terms:
- Submission: the code + model files (
resources/) I push withcrunch push. - Models: what’s inside a submission (trained weights, blend weights, the
train()/infer()code). - Run: one execution of a submission on the platform. The selected run is the one the project is scored with.
As I understand it, the models in a submission can’t be edited in place. To change them I push an updated submission, create a run of it, and select that run. That new run first re-predicts the ~300-moon backfill (unscored), then predicts one live moon per week, and each live prediction is scored ~5 weeks later (week 1 → week 6).
Example: run A (from submission version 1) is selected and predicts live moons for weeks 1–3. In week 4 I change the models, push version 2, and select its run B.
- Old run’s pending predictions: Are run A’s live predictions for weeks 1–3 still scored and shown on the leaderboard when they resolve (weeks 6–8), even though A is no longer selected?
- Only the selected run? Or is only the currently selected run scored, so the project shows no score until run B’s first live moon resolves (week 9)?
- Rewards: Is a week’s payout based on the run that made that week’s prediction, or on the run selected when the score is published?
- Switching back: If I select run A again later (it kept running), are its pending predictions scored again, or does it start over?
- Changing models without a new run: Is there a way to update the models of the submission behind the selected run (e.g. replace files in
resources/) without creating a new run and redoing the backfill?
Background: I want to know how often I can change the models without leaving weeks with no score.
Thanks!