I would appreciate it if you could provide a rationale for why the peer review evaluation in Crunch 3 should be considered reliable and effective. I have two primary concerns:
Requirement for Expert-Level Knowledge: The peer review process appears to assume that participants possess expert-level knowledge, particularly in the development of gene panels and the related algorithmic processes. Given that Crunch 3 is open to a wide range of participants, many of whom may not have specialized expertise in these areas, it seems unrealistic to expect that all reviewers will have the necessary background to evaluate the work accurately. How can we ensure that non-experts are able to provide meaningful feedback, especially when the tasks demand a deep understanding of technical and scientific concepts?
Conflict of Interest: There seems to be a built-in conflict of interest in the peer review scoring system. If participants are incentivized to rate others highly to receive positive feedback themselves, there may be little motivation to provide honest or critical assessments. This creates a potential for inflated scores that do not accurately reflect the quality of the work. Additionally, if ranking in the system is directly affected by peer review scores, what prevents participants from strategically downgrading others’ work to improve their own ranking?
I think those are good points, but also, most participants in the latest milestone already have some background or familiarity with this field. Incorporating peer review alongside our evaluation of top discriminative genes would actually complement our methodology by adding qualitative insights rather than undermining it.
@raghvendramall They mentioned they are going to use a quantitative and objective approach to rank submissions based on my understanding.
Quoting from broad 3 page:
“Classification Accuracy: We’ll use your top 50 genes to train a model that distinguishes between dysplasia and noncancerous mucosa. The better your genes help the model correctly identify these regions, the higher your accuracy score will be. This is the main factor in determining your ranking.”
Unfortunately, there are now additional concerns following the decision to share the top 50 gene list with reviewers. Doesn’t this risk introducing bias - where reviewers may prioritize submissions that align closely with their own gene lists? This could undermine the objectivity of the evaluation process and affect the diversity of selected entries.
To help the reviewer provide meaningful feedback, we have added information/aspects suggesting to cover in the review platform (the team, already did this). For the conflict of interest or bias, we will evaluate the reviewer reports.
Since the gene list is one of the major concrete results, showing them during the reviewing process is valuable.
In my opinion the Evaluation step on page 18 of the challenge doc should determine the final winner. This is under the general principle that the cards speak. All final submissions, whether or not they are high on peer review and Crunch 1 or 2 scores, should have their gene lists assessed against the final panel chosen. This is to rule out possibility that the “right” panel was overlooked due to peer review bias or Crunch 1/2 bias. The scoring rules for final Evaluation seem a bit astrological: "For the top 5 teams (either by Route 1 or Route 2 above), these 50 genes will be the top 50 genes in their ranking; for other teams, these top 50 genes will likely be ranked differently* . Then “We will compute an overall ranking by weighting the cell classification and diversity rankings. The ranking will be mainly determined by the classification accuracy as described above and supplemented by diversity rankings.” The 3 bold phrases are weasel words (weasel phrases?). Before they do the ranking, these should be made more specific. In other words, the scoring function should be something that I can feed into an LLM and get out a reliable piece of Python code to compute the score. For example, here is a run through Perplexity of page 18: https://www.perplexity.ai/search/write-a-python-routine-that-im-6hpgCg4ORE.cjwnp9RqClQ