Based on spearman per cell, amaris is better and if we take average of spearman per cell and spearman per gene, amaris is still better, then how is ranking being determined?
The rank is the average rank of both metrics: competitions/competitions/broad-2/scoring/leaderboard.py at ca86b49dc0481cc795fe16ee7c4b5b5614986394 · crunchdao/competitions · GitHub
@enzo Previously the ranking was stated to be per cell spearman and now based on a small set of 20 genes, we also have to perform well (per gene spearman). If someone over optimizes for these 20 genes and attains low generalization across the 2000 genes (per gene spearman), then how will this be accounted for (as we don’t have measurements from spatial transcriptomics for this entire set). Is the current metric fair, to take combination of both spearman per cell and per gene?
We don’t know these 20 genes, so it is pretty hard to overfit on these 20 genes accounting there are 2000 genes (1%).
Got ur point but the ranking metric is still very weird, like u can be the best on per cell spearman but then if are performing average on per gene spearman, you are ranked down as the ranking is based on the position with respect to other crunchers.
The metric should be absolute in terms of quantitative performance on the criteria and not based on ur rank w.r.t. others, a point raised earlier also
Btw @tarandros thanks a lot for your base code, it was a greater starter point
1 Like
For almost every data science problem, there is no optimal metric, making it nearly a nightmare to choose one. Therefore, the metric is only fair at certain points…
On my side, I’m trying to focus the most on model building and minimize the time spent optimizing the metric. In the hypothesis that if a model is better, it should perform better on all possible metrics.