Would that simply stop the train function and move to infer.
Or somehow that would nullify the predictions for the whole week?
Thanks!
You are gonna be disqualified.
As one missed week because of a crash or a timeout will give you a null score.
And as you can imagine, null is gonna propagate in your average.
1 Like
Okay, I guess maybe null should be -1 instead… But I suppose with only 32 dates, getting a -1 for 3 dates would be as good as disqualification.
Is it safe to assume that the AWS machines during the OOS will be the same (or better, since I saw that the RAM is 128 in OOS, and 64 in the LB phase) as during the LB phase?
thanks @enzo
@xgilbert could you confirm?
Sorry its a mistake in the documentation.
The runners will be exactly the same, only the data will be different.
The RAM will be 64gb for CPU and 128 for GPU.
1 Like
I made a mistake and I updated the documentation about the out-of-sample runtime quota:
Its not 16h per 3 dates, but 23h for the whole 36 dates now.
Very sorry about that.
To make such a drastic change one day before the submission deadline in the competition is completely unprofessional and ridiculous… Especially when I had already pointed out an error in the documentation previously.
People (like me) have worked within the rules and spent a lot of time to make sure it would fit within such constraints.
I apologize if I sound rude, but this is a very big deal.
Its actually the documentation that was wrong all along and me too because of it.
The real value has been said in the AMA #4 at 19m56s:
Since you managed to run under the 20h limit, you should not be impacted anyway but still I can understand your disappointment.
EDIT: Typos
Maybe that;s just me but I think the official documentation is more important than to get the info at 19m56 in AMA #4…
Well I ran something under 20 hours but it’s a very different setting when you go to OOS, I am definitely impacted.
I confirm about disqualification.
A -1 score could get you an advantage as it could be higher than some competitor’s score in some cases.
isn’t -1 the minimum value rank spearman can take?
You are right; Since the spearman is expressed in % in the scores, I was assuming you were talking about a percentage.
Anyway, we want to reproduce as closely as possible the behavior of the submission phase, so we will not change this behavior.
As Enzo said in the discord, you can get the time of computation you have left and stop your training based on this.
This sounds amazing as I found it very much useful
and informative to be honest. Also, I have gone through this post which
definitely helped me out a lot as a new member I am looking forward for more
such discussions.