MedProb challenges the assumption that medical visual question answering requires large models or complex pipelines. It predicts answers directly from frozen vision-language model (VLM) representations, avoiding free-text generation.
The framework was tested across datasets including PATH-VQA, SLAKE, and VQA-RAD. It recovers more answer-relevant signals than prompting and performs better than medical VLMs and agentic systems. Probing reduces the performance gap between small and large models compared to prompting.
Analysis shows that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation suggests. Medical adaptation does not consistently improve linear decodability across matched general-purpose and medical VLM pairs. The approach can be extended to open-ended generation using rejection sampling.
This method offers a simpler, more effective way to evaluate and utilize VLMs for medical question answering, emphasizing the importance of probing over fine-tuning or complex pipelines.
Source: https://arxiv.org/abs/2609.04336