Hi, could you again make the leaderboard live? It worked quite well in crunch 1? Also it will make the issues described here more obvious quicker The current scoring is unstable
Here is a rephrased response from the Broad team:
As you previously indicated, doing a live ranking could potentially enable individuals to “reverse engineer” the dataset.
We are concerned about overfitting due to the relatively small dataset, which consists of only 20 genes.
Initially, Crunch 2 was not designed to include checkpoints. However, we acknowledge the value of providing feedback on model performance.
We determined that two checkpoints per week represented a reasonable compromise to mitigate the risk of overfitting.
I didn’t say that “live ranking would lead to reverse engineering” - the issue is the concept behind the current ranking scheme. Live or 2-times a week, the issue is still there.
Also about your 20 genes - this is why you have validation and test set, right?
Right now, if most participants optimize for metric A while only a few focus on metric B, those optimizing for metric B will be favored due to lower competition and higher ranks—and vice versa. This introduces an element of luck, as a model’s ranking depends on the choices of the majority rather than its actual performance. Moreover, the rankings will fluctuate constantly based on which metric happens to be more favorable for the larger group of players at any given time.