Hi, would it be possible to trigger the first checkpoint validation ourselves in the coming week? Or does it happen automatically only once today? Is there a way to make it run continuously and have a “live” leaderboard? Thanks in advance!
Hello,
The leaderboard will only update at checkpoints and there is no way to have a trigger per user request or a continuous leaderboard like we always do.
Broad Institude wants to make sure that overfiting is impossible by only “offering” to score 2 times.
Hi,
-
would it be possible to make the scoring results public? This would help comparing and get an idea of the level of improvement needed to be done.
-
Also, I see that all submissions per model were graded, which maybe wasn’t very clearly communicated? Would it be possible to select the submission with the highest test score to count during the test phase at the end? So theoretically someone could be submitting 5 different submissions each day and still overfit on the validation/test.
-
How would the finalists be defined at the end? Per model (i.e., one user could win up to 3 prizes) or per user/team by taking the maximum performance across models and submissions?
-
Regarding the live leaderboard and given that currently all submission per model were evaluated, maybe having a small “evaluation budget” would make more sense and make the competition more interesting and active, eventually leading to better modelling strategies.
- We just released the leaderboard. Leaking the result on your dashboards was an accident of mine
- This is the point of the selection system. You have to choose a Run that will be used to score you at the final checkpoint.
- It is per model, and if they are in a team, other rules strictly apply: Rewards | CrunchDAO Docs V3
- We cannot allow a user based solution as it could be abused. But thanks to your (yes you @many-kalin) many suggestions, we have doubled the number of checkpoints. They are now every 2 weeks.
1 Like
If you are going to run a checkpoint every 2 weeks, that would be today (16.12.2024) or tomorrow. Do you agree ?
Today at 5 PM UTC
it will likely take 2 hours.
Isn’t it December 16th (Eastern Time 17:59)?
That will be the case for the final deadline.
But 6 PM EST is Midnight Paris Time, so it is a bit late for us.
Ok, the documentation is indeed saying EST…
I see but It was announced on the main HUB page (also on discord)… Can you postpone it for the morning?
Sorry, I just edited my previous message.
I will do it at the previously mentionned time.
1 Like
Hi, I submitted a new model for checkpoint on Dec.16 but the leaderboard still uses my previous submission. I’m pretty sure I’ve selected the new one, RUN ID #26386 and PREDICTION ID #22483. Can you please check this for me and update the leaderboard if possible? Thank you in advance!
We are gonna announce the leaderboard soon.
Oh, I thought it was already updated as there was some changes. Never mind then. Thank you!