Qwen3 VL 235B A22B Thinking

Leaderboards

Where Qwen3 VL 235B A22B Thinking sits among the top 20 in each category.

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Test results

Every test Qwen3 VL 235B A22B Thinking has a published result on, out of 100%, with its rank among the models that took it.

• MMLU-Pro

  #32 / 142

  Broad knowledge and reasoning across 14 subjects, ten answer choices each.

  83.8%

  Self-reported

• AIME 2025

  #51 / 123

  Competition maths problems from the 2025 American Invitational Mathematics Exam.

  89.7%

  Self-reported

• Humanity's Last Exam

  #89 / 108

  Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions

  13.6%

  Self-reported · In the text score· In the vision score

• MMLU

  #6 / 103

  Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains

  90.6%

  Self-reported

• MMMU-Pro

  #42 / 73

  College-level questions that need both an image and text to answer.

  69.3%

  Self-reported · In the vision score

• IFEval

  #28 / 70

  Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints

  88.2%

  Self-reported

• LiveCodeBench v6

  #40 / 67

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  70.1%

  Self-reported

• CharXiv-R

  #45 / 60

  Reading and reasoning about charts from scientific papers.

  66.1%

  Self-reported

• MMLU-Redux

  #8 / 50

  An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.

  93.7%

  Self-reported

• SimpleQA

  #18 / 47

  SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.

  44.4%

  Self-reported

• MathVision

  #20 / 38

  MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.

  74.6%

  Self-reported

• SuperGPQA

  #15 / 37

  SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.

  64.3%

  Self-reported

• AI2D

  #14 / 35

  AI2D is a dataset of 4,903 illustrative diagrams from grade school natural sciences (such as food webs, human physiology, and life cycles) with over 15,000 multiple choice questions and answers. The benchmark evaluates diagram understanding and visual reasoning capabilities, requiring models to interpret diagrammatic elements, relationships, and structure to answer questions about scientific concepts represented in visual form.

  89.2%

  Self-reported

• MMLU-ProX

  #10 / 32

  Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.

  80.6%

  Self-reported

• RealWorldQA

  #15 / 32

  RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts.

  81.3%

  Self-reported

• Include

  #9 / 31

  Include benchmark - specific documentation not found in official sources

  80.0%

  Self-reported

• LVBench

  #19 / 31

  LVBench is an extreme long video understanding benchmark designed to evaluate multimodal models on videos up to two hours in duration. It contains 6 major categories and 21 subcategories, with videos averaging five times longer than existing datasets. The benchmark addresses applications requiring comprehension of extremely long videos.

  63.6%

  Self-reported

• ERQA

  #19 / 29

  Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions

  52.5%

  Self-reported

• HMMT25

  #15 / 28

  Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds

  77.4%

  Self-reported

• VideoMMMU

  #19 / 28

  Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.

  80.0%

  Self-reported

• MMStar

  #9 / 27

  MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans to evaluate 6 core capabilities and 18 detailed axes. The benchmark addresses issues of visual content unnecessity and unintentional data leakage in existing multimodal evaluations.

  78.7%

  Self-reported

• ScreenSpot Pro

  #16 / 27

  ScreenSpot-Pro is a novel GUI grounding benchmark designed to rigorously evaluate the grounding capabilities of multimodal large language models (MLLMs) in professional high-resolution computing environments. The benchmark comprises 1,581 instructions across 23 applications spanning 5 industries and 3 operating systems, featuring authentic high-resolution images from professional domains with expert annotations. Unlike previous benchmarks that focus on cropped screenshots in consumer applications, ScreenSpot-Pro addresses the complexity and diversity of real-world professional software scenarios, revealing significant performance gaps in current MLLM GUI perception capabilities.

  61.8%

  Self-reported

• MathVista-Mini

  #8 / 26

  MathVista-Mini is a smaller version of the MathVista benchmark that evaluates mathematical reasoning in visual contexts. It consists of examples derived from multimodal datasets involving mathematics, combining challenges from diverse mathematical and visual tasks to assess foundation models' ability to solve problems requiring both visual understanding and mathematical reasoning.

  85.8%

  Self-reported

• Multi-IF

  #5 / 25

  Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.

  79.1%

  Self-reported

• OCRBench

  #13 / 24

  OCRBench: Comprehensive evaluation benchmark for assessing Optical Character Recognition (OCR) capabilities in Large Multimodal Models across text recognition, scene text VQA, and document understanding tasks

  87.5%

  Self-reported

• OSWorld

  #12 / 21

  OSWorld: The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS with 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows

  38.1%

  Self-reported

• BFCL-v3

  #6 / 20

  Berkeley Function Calling Leaderboard v3 (BFCL-v3) is an advanced benchmark that evaluates large language models' function calling capabilities through multi-turn and multi-step interactions. It introduces extended conversational exchanges where models must retain contextual information across turns and execute multiple internal function calls for complex user requests. The benchmark includes 1000 test cases across domains like vehicle control, trading bots, travel booking, and file system management, using state-based evaluation to verify both system state changes and execution path correctness.

  71.9%

  Self-reported

• Hallusion Bench

  #6 / 20

  A comprehensive benchmark designed to evaluate image-context reasoning in large visual-language models (LVLMs) by challenging models with 346 images and 1,129 carefully crafted questions to assess language hallucination and visual illusion

  66.7%

  Self-reported

• MMBench-V1.1

  #7 / 20

  Version 1.1 of MMBench, an improved bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks.

  90.6%

  Self-reported

• CC-OCR

  #5 / 18

  A comprehensive OCR benchmark for evaluating Large Multimodal Models (LMMs) in literacy. Comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. Contains 39 subsets with 7,058 fully annotated images, 41% sourced from real applications. Tests capabilities including text grounding, multi-orientation text recognition, and detecting hallucination/repetition across diverse visual challenges.

  81.5%

  Self-reported

• BLINK

  #11 / 17

  BLINK: Multimodal Large Language Models Can See but Not Perceive. A benchmark for multimodal language models focusing on core visual perception abilities. Reformats 14 classic computer vision tasks into 3,807 multiple-choice questions paired with single or multiple images and visual prompting. Tasks include relative depth estimation, visual correspondence, forensics detection, multi-view reasoning, counting, object localization, and spatial reasoning that humans can solve 'within a blink'.

  67.1%

  Self-reported

• MM-MT-Bench

  #2 / 17

  A multi-turn LLM-as-a-judge evaluation benchmark for testing multimodal instruction-tuned models' ability to follow user instructions in multi-turn dialogues and answer open-ended questions in a zero-shot manner.

  8.5%

  Self-reported

• SimpleVQA

  #12 / 17

  SimpleVQA is a visual question answering benchmark focused on simple queries.

  0.6%

  Self-reported

• ODinW

  #10 / 16

  Object Detection in the Wild (ODinW) benchmark for evaluating object detection models' task-level transfer ability across diverse real-world datasets in terms of prediction accuracy and adaptation efficiency

  43.2%

  Self-reported

• ScreenSpot

  #3 / 16

  ScreenSpot is the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. The dataset comprises over 1,200 instructions from iOS, Android, macOS, Windows and Web environments, along with annotated element types (text and icon/widget), designed to evaluate visual GUI agents' ability to accurately locate screen elements based on natural language instructions.

  95.4%

  Self-reported

• WritingBench

  #3 / 16

  A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions.

  86.7%

  Self-reported

• LiveBench 20241125

  #1 / 15

  LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.

  79.6%

  Self-reported

• OCRBench-V2 (en)

  #5 / 15

  OCRBench v2 English subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with English text content

  66.8%

  Self-reported

• ZEROBench

  #15 / 15

  ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks.

  0.0%

  Self-reported

• Creative Writing v3

  #5 / 14

  EQ-Bench Creative Writing v3 is an LLM-judged creative writing benchmark that evaluates models across 32 writing prompts with 3 iterations per prompt. Uses a hybrid scoring system combining rubric assessment and Elo ratings through pairwise comparisons. Challenges models in areas like humor, romance, spatial awareness, and unique perspectives to assess emotional intelligence and creative writing capabilities.

  85.7%

  Self-reported

• MuirBench

  #2 / 14

  A comprehensive benchmark for robust multi-image understanding capabilities of multimodal LLMs. Consists of 12 diverse multi-image tasks involving 10 categories of multi-image relations (e.g., multiview, temporal relations, narrative, complementary). Comprises 11,264 images and 2,600 multiple-choice questions created in a pairwise manner, where each standard instance is paired with an unanswerable variant for reliable assessment.

  80.1%

  Self-reported

• CharadesSTA

  #2 / 12

  Charades-STA is a benchmark dataset for temporal activity localization via language queries, extending the Charades dataset with sentence temporal annotations. It contains 12,408 training and 3,720 testing segment-sentence pairs from videos with natural language descriptions and precise temporal boundaries for localizing activities based on language queries.

  63.5%

  Self-reported

• InfoVQAtest

  #2 / 12

  InfoVQA test set with infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills

  89.5%

  Self-reported

• DocVQAtest

  #3 / 11

  DocVQA is a Visual Question Answering benchmark on document images containing 50,000 questions defined on 12,000+ document images. The benchmark focuses on understanding document structure and content to answer questions about various document types including letters, memos, notes, and reports from the UCSF Industry Documents Library.

  96.5%

  Self-reported

• OCRBench-V2 (zh)

  #1 / 11

  OCRBench v2 Chinese subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with Chinese text content

  63.5%

  Self-reported

• MLVU

  #9 / 10

  A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization.

  83.8%

  Self-reported

• VideoMME w/o sub.

  #6 / 10

  Video-MME is a comprehensive evaluation benchmark for multi-modal large language models in video analysis. It features 900 videos across 6 primary visual domains with 30 subfields, ranging from 11 seconds to 1 hour in duration, with 2,700 question-answer pairs. The benchmark evaluates MLLMs' capabilities in processing sequential visual data and multi-modal content including video frames, subtitles, and audio.

  79.0%

  Self-reported

• CountBench

  #8 / 9

  CountBench evaluates object counting capabilities in visual understanding.

  0.9%

  Self-reported

• RefCOCO-avg

  #3 / 9

  RefCOCO-avg measures object grounding accuracy averaged across RefCOCO, RefCOCO+, and RefCOCOg benchmarks.

  0.9%

  Self-reported

• ZebraLogic

  #1 / 9

  ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.

  97.3%

  Self-reported

• EmbSpatialBench

  #3 / 8

  EmbSpatialBench evaluates embodied spatial understanding and reasoning capabilities.

  0.8%

  Self-reported

• RefSpatialBench

  #3 / 8

  RefSpatialBench evaluates spatial reference understanding and grounding.

  0.7%

  Self-reported

• MMLongBench-Doc

  #6 / 7

  MMLongBench-Doc evaluates long document understanding capabilities in vision-language models.

  0.6%

  Self-reported

• ZEROBench-Sub

  #6 / 6

  ZEROBench-Sub is a subset of the ZEROBench benchmark.

  0.3%

  Self-reported

• Hypersim

  #4 / 4

  Hypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes.

  0.1%

  Self-reported

• MMMUval

  #1 / 4

  Validation set for MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) benchmark, designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.

  80.6%

  Self-reported

• SUNRGBD

  #3 / 4

  SUNRGBD evaluates RGB-D scene understanding and 3D grounding capabilities.

  0.3%

  Self-reported

• VisuLogic

  #4 / 4

  VisuLogic evaluates logical reasoning capabilities in visual contexts.

  0.3%

  Self-reported

• Design2Code

  #2 / 2

  Design2Code evaluates the ability to generate code (HTML/CSS/JS) from visual designs.

  0.9%

  Unverified

• ARKitScenes

  #1 / 1

  ARKitScenes evaluates 3D scene understanding and spatial reasoning in AR/VR contexts.

  0.5%

  Unverified

• MIABench

  #1 / 1

  MIABench evaluates multimodal instruction alignment and following capabilities.

  0.9%

  Unverified

• MathVerse-Mini

  #1 / 1

  MathVerse-Mini is a subset of the MathVerse benchmark for evaluating math reasoning capabilities in vision-language models.

  0.9%

  Unverified

• OSWorld-G

  #1 / 1

  OSWorld-G (Grounding) evaluates screenshot grounding accuracy for OS automation tasks.

  0.7%

  Unverified

• Objectron

  #1 / 1

  Objectron evaluates 3D object detection and pose estimation capabilities.

  0.7%

  Unverified

• RoboSpatialHome

  #1 / 1

  RoboSpatialHome evaluates spatial understanding for robotic home navigation and manipulation.

  0.7%

  Unverified

• SIFO

  #1 / 1

  SIFO (Simple Instruction Following) evaluates how well language models follow simple, explicit instructions. It tests fundamental instruction-following capabilities across various task types.

  0.8%

  Unverified

• SIFO-Multiturn

  #1 / 1

  SIFO-Multiturn evaluates instruction following capabilities in multi-turn conversational settings, testing how well models maintain context and follow instructions across multiple exchanges.

  0.7%

  Unverified