U2-BENCH: Revolutionizing Ultrasound AI Evaluation Standards

U2-BENCH: Revolutionizing Ultrasound AI Evaluation Standards

Dolphin AI, a Chinese startup specializing in ultrasound-specific medical intelligence, has introduced U2-BENCH, a groundbreaking evaluation standard for multimodal ultrasound AI. This research, accepted by ICLR 2026, aims to bridge the gap between general AI capabilities and specialized clinical ultrasound requirements. Ultrasound imaging is crucial in healthcare, but automated ultrasound image understanding faces challenges like variability and complex spatial relationships. U2-BENCH is the first benchmark to evaluate LVLM capabilities in ultrasound, covering classification, detection, regression, and text generation tasks. It includes data from 40 datasets and 50 clinical use cases to ensure accurate evaluation results. The benchmark organizes ultrasound understanding into four capability levels and eight specific tasks. An experimental validation of 23 vision-language models on U2-BENCH showed that closed-source models like Dolphin-V1 outperformed open-source models. There is a gap between recognition and reasoning tasks, with models struggling in spatial-related tasks and report generation. The study concluded that scaling alone is not the answer and domain-specific training is more effective than expanding model parameters. U2-BENCH signifies a shift towards all-encompassing foundation models in ultrasound AI and is expected to expand to include dynamic video understanding and long-range embodied perception.