These earlier comparisons retain their original outputs and ratings. They are a separate selection; do not combine the case counts into an independent unseen test.
The same rubric for both answers
What we rate, and why.
Each criterion is marked from 0 to 10. The overall answer score is the weighted sum shown below, rounded to one decimal. A confident diagnosis earns credit when the evidence supports it; additional caution does not automatically improve a mark.
Rubric · five criteria · weights total 100%| Criterion | Weight | What earns credit |
|---|
| Diagnosis and case fit | 25% | Does the leading explanation account for the important findings and material alternatives? |
|---|
| Interpretation of tests and experiments | 25% | Does the answer identify what the actual specimen, controls and measurement establish? |
|---|
| Strength of the claims | 20% | Are firm conclusions stated when justified, without turning an association or plausible mechanism into proof? |
|---|
| Decisions and next steps | 15% | Does the answer identify the useful next investigation and make time-sensitive priorities visible? |
|---|
| Clarity and teaching | 15% | Is the mechanism understandable, focused and easy to connect to the case? |
|---|
Score anchors and assessment method
- 0–2
- Absent or fundamentally wrong
- 3–4
- Major errors or missing reasoning
- 5–6
- Useful core answer with a material error or omission
- 7–8
- Strong answer with specific shortcomings
- 9
- Very strong; only small shortcomings
- 10
- No material shortcoming identified for this criterion in the supplied case
How these ratings were produced
This is MontMedical's AI-assisted editorial assessment, dated 16 November 2026. We read the full retained answer texts and checked the material experimental criticisms against the linked sources. The rubric was applied retrospectively, with the product names visible; there was no independent blinded panel.
The eight synthetic examples were selected during development. MontMedical material consists of retained diagnostic/further-evidence sections; OpenEvidence answers often include additional management discussion. We score the supplied text, so a missing next step can lower a mark even if a different stage of the app might supply it.
The case fingerprints match, but exact prompts submitted to OpenEvidence and complete original run settings are not independently retained. Scores describe these historical answers, not current product-wide accuracy. We did not audit every reference in both outputs or measure patient outcomes, latency or cost.
Any important omission remains visible alongside the total. A weighted average does not cancel a missed urgent priority.
Selected retained answers / September–October 2026
The ratings, case by case.
Both tools identify the leading diagnosis in most examples. The differences come from the interpretation, the confidence of the claim and the usefulness of the next step.
Weighted editorial scores out of 10 · exact passages and criterion marks on each case page| Case | OpenEvidence | MontMedical | Principal difference |
|---|
| Familial prothrombin disorder | 7.4 | 8.7 | MontMedical is stronger on experimental interpretation. |
|---|
| Hereditary xerocytosis | 7.8 | 8.4 | MontMedical is stronger on the gene-versus-pathway distinction. |
|---|
| Quebec platelet disorder | 8.1 | 8.4 | MontMedical has the better-calibrated diagnostic wording. |
|---|
| NCF4 / p40phox deficiency | 8.4 | 8.4 | Different strengths; the weighted scores are level. |
|---|
| CANOMAD | 8.2 | 8.1 | OpenEvidence is stronger on urgency; MontMedical is stronger on the assay boundaries. |
|---|
| ISCU-type myopathy | 8.2 | 7.6 | OpenEvidence gives the more useful tissue-specific explanation. |
|---|
| Glucocorticoid resistance | 8.2 | 8.2 | Comparable overall: OpenEvidence leads on presentation; MontMedical on interpretation. |
|---|
| Lysinuric protein intolerance | 8.2 | 7.8 | OpenEvidence explains the physiology and next steps more effectively. |
|---|
| Mean of these eight answers | 8.0 | 8.2 | Selected examples; no population-wide accuracy estimate. |
|---|
Ratings are editorial judgments by the commercially interested publisher, MontMedical. A close numerical difference is not a statistically established performance difference. Read the rubric and method before interpreting the totals.
Published research / separate evidence
OpenEvidence vs frontier LLMs: what published comparisons show
Two 2026 published studies put frontier models ahead of OpenEvidence on defined tasks. See the results, a contrasting clinician-rated preprint, and what they mean.
Read the studies and their limitations →Final assessment
Where OpenEvidence is weaker than MontMedical.
The strongest supported weakness in these examples is overinterpreting a useful finding. Both tools often identify the same leading disease. The difference is whether the explanation preserves the gap between the observed result and the mechanism it claims.
- Localising an abnormality too precisely. In the prothrombin case, the abnormality follows a patient-derived preparation. OpenEvidence assigns it to the molecule itself and treats the long peptide-cleavage tail as the characteristic inhibition defect. MontMedical separates preparation-level transfer, substrate cleavage and direct inhibition kinetics.
- Inferring the causal gene from a pathway response. In hereditary xerocytosis, both answers identify the correct disease group. OpenEvidence's later blocker discussion moves towards KCNN4-associated disease; MontMedical explains why a PIEZO1 defect can activate the same downstream Gardos channel.
- Overstating specificity or exclusion. The platelet case calls an aggregation clue specific and recommends removing a historical diagnosis. The CANOMAD answer treats Bing–Neel syndrome as effectively excluded. MontMedical more precisely distinguishes a strong leading interpretation from a categorical exclusion.
Where OpenEvidence is stronger
OpenEvidence gives the more connected physiological explanation in LPI and the more useful tissue-specific account in ISCU myopathy. It also makes the CANOMAD respiratory deterioration urgent, where the retained MontMedical section leaves the priority buried. Those advantages are reflected in the ratings.
What MontMedical needs to keep improving
Put urgent implications where readers can see them, make the next diagnostic step explicit, and replace dense result lists with a short explanation of the mechanism. Precision should make the answer easier to act on, rather than merely longer.
Why favour MontMedical's approach? Its clearest advantage here is disciplined interpretation: explain what the experiment establishes, retain the strong leading diagnosis and identify the test that would resolve the remaining causal link. The selected historical ratings show that advantage on particular questions; they do not establish universal superiority or clinical efficacy.