Stanford MARVL
Advance visual perception in biomedical multimodal models.
01About
The MMBU Challenge aims to improve how multimodal models perceive biomedical images across modalities, scales, and anatomical regions.
Today’s multimodal large language models (MLLMs) can produce seemingly impressive biomedical answers. But beneath these capabilities lies a fundamental weakness: they may not even understand what they are looking at. In biomedical MLLMs, visual perception remains the primary bottleneck for reliable downstream reasoning. So we built a challenge to put biomedical visual understanding to the test.
02Tracks
A checkpoint can enter only one track. Models over 27B active parameters need an API key for evaluation.
Best overall performance.
Submit the strongest biomedical perception model. There is no size limit. Models with more than 27B active parameters must be available for evaluation through an API key.
Improvement over a disclosed base model.
Adapt a pretrained model toward medicine through mid-training or post-training. Ranking rewards the value added by that adaptation, not the strength of the starting checkpoint. Models with more than 27B active parameters need an API key.
Maximum perception with minimal computation.
Pretrain or adapt a small biomedical model. This track is for parameter-efficient models that keep strong biomedical perception. Disclose the number of active parameters.
03Resources
Teams receive a public development set, weekly office hours with the organizers, special credits from Anthropic and GXL, and cash prizes. The top entry in each track will also be invited to contribute to the MMBU Challenge technical report.
04Sponsors
Official sponsors of the MMBU Challenge. Collaborator institutions are named in the FAQ, in text only.
05FAQ
MMBU evaluates open-ended visual question answering on a peer-reviewed biomedical benchmark. Models must recognize, localize, and characterize visual features — then identify imaging metadata in a separate pass, so they cannot shortcut the answer.
Teams must apply and be accepted before competing. Please apply to participate using the LUMA event page.
Two independent passes on the hidden set. Pass 1 is the task answer (T). Pass 2 is biomedical context (C): modality, submodality, specimen, body part, stain, and domain. The shared base score is 0.8T + 0.2C. Each track then applies its own rule. Ties break on T. Sonnet-5 is the judge.
Single end-to-end VLM. No ensembles, retrieval, web search, or extra tools at eval. Official harness only, one pass per example. Models under 27B (and all of Track 3) must be public on Hugging Face. Disclose training data. Do not train on hidden evaluation examples. Full rules are in the challenge brief.
A public development set will be released for local evaluation. Final rankings use a private held-out set of open-ended VQA only. Dev-set scores will not match the final leaderboard.
Stanford MARVL, led by Serena Yeung-Levy. Primary organizers: Ryan D’Cunha, Alejandro Lozano, Akira Nishii, and Maximilian Rokuss. Sponsors may participate because only organizers see the private set.
Individual researchers advising on design — not institutional partners, and not sponsors. Affiliations for identification only: Stanford, Cambridge, University of Washington, MD Anderson, Microsoft AI, Google DeepMind, and Apple. Their logos are not used on this site.
