Hello,
Could you clarify the purpose of these checkpoints? (“To avoid overfitting, Crunch will score all submitted models on a private dataset during two checkpoints.”) Is it mandatory to submit for each checkpoint, or is it an optional external validation? Will we have access to this validation data?
Thank you in advance!
Two checkpoints are scheduled: the first on November 30, 2024, and the second on December 30.
You will need to predict the validation and test cells. The validation cell will be scored at both the first and second checkpoints. You will then have the opportunity to adjust your model and submit a new version after each checkpoint. Finally, the test cell will be scored at the close of Crunch 1 on January 31, 2025, which will determine the final score for the competition Crunch 1
Thanks for answering. Are we going to have access/be able to download the validation data? how are we going to access the checkpoint validation data? Is it mandatory to already have a submission before the checkpoints on November 30th?
Are we going to have access to/download the validation data? How will we access the checkpoint validation data?
The validation data won’t be provided; it will remain private, as per our client’s request. However, the score you receive will let you assess your model’s performance. That said, we’re bringing this up with the Broad team to see if there’s any possibility of releasing the validation data after the second checkpoint.
Is it mandatory to already have a submission before the checkpoints on November 30th?
It’s not mandatory. You’ll simply miss an opportunity to evaluate your model if you choose not to submit before then.