How is crunch 3 scored? Is it just by the top 50 genes of the list?
I recommend you read the documentation:
But to make it short, once the part 3 ended, a peer review phase will start.
The top 5 submission will be used to select 250 genes. Then will be combined with 250 other genes from the top 5 of part 2 (if they participated to part 3).
Then the broad team will order a Xenium gene panel on those 500 genes allowing us to then score everyone on the panel’s results.
I understand this part. But how will the results from the actual panel be used to evaluate? It seems pretty arbitrary at the moment.
The Xenium gene panel can only detect 460 genes at a time. Hence the selection.
We will only be able to score prediction once we have the result.
I understand the selection. What I am curious is on whats the ground truth here? Log fold change based on the actual Xenium panel?
I saw in the review part. Classification Accuracy : We’ll use your top 50 genes to train a model that distinguishes between dysplasia and noncancerous mucosa. The better your genes help the model correctly identify these regions, the higher your accuracy score will be. This is the main factor in determining your ranking.
If this is used as evaluation, whats the machine learning model used? How is it trained or finetuned? Why dont the participant submit a trained model based on their top 50 genes. This seems more fair in my opinion.
At this point we don’t know yet which genes will be evaluated, that is why we ask participant to rank them and then peer review the work of others (via the report.md, not the code).
The top 50 might be a top 70 if the top 5 models have ranked similar genes to the top.
The reports needs to be comprehensive or as stated in the sample notebook can be concise but enough to justify the choice and direction took to reach the final ranking of the 18,615 genes?
The official rules say that it cannot be longer than one page.
Being consise would be great, but you are allowed to add details if you want.
We may penalize people who try to abuse the system.
1 Like
I still don’t understand how the actual panel is going to be used for evaluation. What’s the ground truth? What’s the metric going to be used?
The ground truth is what gonna be the result of the ordered panel.
Regarding the metrics, I do not have any information for now.
Do you know if they have any plans to release the metrics anytime soon ? Now it seems to be random shot.
I also want to clarify on the xenium panel being the ground truth. The ground truth should be a list of ranked genes. So how it being derived from the xenium data? Is it through differntial expression of the actual panel ? Or is it some machine learning model that broad is going to use.