I am testing a frozen Synth tracker under model theoretical-hamster (model ID 17028,
submission 1, deployment 21365). Could you point me to the supported documentation or export for:
Per-round request/return/validation/scoring records, including unique round IDs, asset and
horizon, timestamps, and missing/late/invalid outcomes? Both the visible Runner log and the
downloaded Runner log contain six startup INFO lines, which do not establish forecast coverage.
The exact formula and direction of the UI’s “Normalized CRPS by Parameters” metric, plus
the identity/version of its displayed benchmark and how to obtain matched-round values?
Which scoring version resulted from the August 17 scoring correction/backfill announcement?
I do not assume it is identical to the public SDK evaluator or Gaussian example tracker.
The current Synth payout schedule and eligibility rules, especially what happens to unresolved
forecasts, provisional allocations and finalized-but-unpaid rewards when Model Runner is
switched Off? Is remaining online required at a checkpoint or distribution time?
I am keeping the model unchanged during a bounded trial and want to distinguish forecast
performance, valid-return coverage, allocations and actual payment. Links or a supported export
procedure would be sufficient. Thank you.
We don’t offer this level of data, but all the metrics we offer are available on your metrics page. While the endpoint isn’t documented, you should be able to discover its parameters by playing around with the DevTools.
Please note that an API Key is also required to access them.
Did you print anything? I’m not aware of any issues with the logs themselves.
If you want to automate that part too, you can access it via the API. You can play around with the DevTools again, but this one is much simpler.
We use the crps_ensemble method from properscoring. We then normalize it based on the best and worst scores of the predict call.
We currently do not share everyone’s values that would be required for you to compute it, sorry.
I’m sorry, but I’m not sure what you mean by this. We did have to backfill the scores, but we used the same formula.
They occur every Monday at 12 p.m. UTC on the current leaderboard.
They keep being resolved, and you remain on the leaderboard until there are none left. After that, you will be marked as “absent” and slowly fade off the leaderboard.
Even if your model is offline during distribution time, you will still receive your share of the pot if you are on the leaderboard and eligible.