# Crunch 3 evaluation
**URL:** https://forum.crunchdao.com/t/crunch-3-evaluation/806
**Category:** (ended) Broad Institute Crunch #3
**Created:** [April 4, 2025, 11:33pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806 "2025-04-04T23:33:36Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@royal-chengwei](https://forum.crunchdao.com/u/royal-chengwei)
#### Post date: [April 4, 2025, 11:33pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/1 "2025-04-04T23:33:36Z")
How is crunch 3 scored? Is it just by the top 50 genes of the list?
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 4, 2025, 11:49pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/2 "2025-04-04T23:49:58Z")
I recommend you read the documentation:
> **[Evaluation - Crunch 3 – Dec 9 to Apr 30 – Identifying Gene | CrunchDAO Docs V3](https://docs.crunchdao.com/competitions/competitions/broad-institute-autoimmune-disease/crunch-3#evaluation)**
>
> In Crunch 3, your task is to design a gene panel that best distinguishes dysplasia regions from noncancerous mucosa regions in colon tissue affected by Inflammatory Bowel Disease (IBD). Using provided H&E images annotated by pathologists and...
But to make it short, once the part 3 ended, a peer review phase will start.
The top 5 submission will be used to select 250 genes. Then will be combined with 250 other genes from the top 5 of part 2 (if they participated to part 3).
Then the broad team will order a Xenium gene panel on those 500 genes allowing us to then score everyone on the panel’s results.
---
### Author:  [@royal-chengwei](https://forum.crunchdao.com/u/royal-chengwei)
#### Post date: [April 5, 2025, 7:30am UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/3 "2025-04-05T07:30:42Z")
I understand this part. But how will the results from the actual panel be used to evaluate? It seems pretty arbitrary at the moment.
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 5, 2025, 12:19pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/4 "2025-04-05T12:19:16Z")
The Xenium gene panel can only detect 460 genes at a time. Hence the selection.
We will only be able to score prediction once we have the result.
---
### Author:  [@royal-chengwei](https://forum.crunchdao.com/u/royal-chengwei)
#### Post date: [April 5, 2025, 12:57pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/5 "2025-04-05T12:57:20Z")
I understand the selection. What I am curious is on whats the ground truth here? Log fold change based on the actual Xenium panel?
I saw in the review part. **Classification Accuracy** : We’ll use your top 50 genes to train a model that distinguishes between dysplasia and noncancerous mucosa. The better your genes help the model correctly identify these regions, the higher your accuracy score will be. This is the main factor in determining your ranking.
If this is used as evaluation, whats the machine learning model used? How is it trained or finetuned? Why dont the participant submit a trained model based on their top 50 genes. This seems more fair in my opinion.
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 5, 2025, 1:20pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/6 "2025-04-05T13:20:53Z")
At this point we don’t know yet which genes will be evaluated, that is why we ask participant to rank them and then peer review the work of others (via the report.md, not the code).
The top 50 might be a top 70 if the top 5 models have ranked similar genes to the top.
---
### Author:  [@raghvendramall](https://forum.crunchdao.com/u/raghvendramall)
#### Post date: [April 6, 2025, 9:52am UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/7 "2025-04-06T09:52:55Z")
The reports needs to be comprehensive or as stated in the sample notebook can be concise but enough to justify the choice and direction took to reach the final ranking of the 18,615 genes?
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 6, 2025, 2:11pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/8 "2025-04-06T14:11:30Z")
The official rules say that it cannot be longer than one page.
Being consise would be great, but you are allowed to add details if you want.
We may penalize people who try to abuse the system.
---
### Author:  [@royal-chengwei](https://forum.crunchdao.com/u/royal-chengwei)
#### Post date: [April 11, 2025, 12:51pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/9 "2025-04-11T12:51:32Z")
I still don’t understand how the actual panel is going to be used for evaluation. What’s the ground truth? What’s the metric going to be used?
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 11, 2025, 1:07pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/10 "2025-04-11T13:07:09Z")
The ground truth is what gonna be the result of the ordered panel.
Regarding the metrics, I do not have any information for now.
---
### Author:  [@royal-chengwei](https://forum.crunchdao.com/u/royal-chengwei)
#### Post date: [April 11, 2025, 1:22pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/11 "2025-04-11T13:22:55Z")
Do you know if they have any plans to release the metrics anytime soon ? Now it seems to be random shot.
I also want to clarify on the xenium panel being the ground truth. The ground truth should be a list of ranked genes. So how it being derived from the xenium data? Is it through differntial expression of the actual panel ? Or is it some machine learning model that broad is going to use.
---
### Author:  [@enzo](https://forum.crunchdao.com/u/enzo)
#### Post date: [April 28, 2025, 3:09pm UTC](https://forum.crunchdao.com/t/crunch-3-evaluation/806/12 "2025-04-28T15:09:19Z")
The bread team answered:
> The ground truth or metric for evaluation will not be decided before we run the Xenium and define some features to quantify and compare. But roughly as you mentioned, we will do some differential expression analysis comparing regions with different pathological features.