Qwen3.5-27B

Leaderboards

Where Qwen3.5-27B sits among the top 20 in each category.

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Test results

Every test Qwen3.5-27B has a published result on, out of 100%, with its rank among the models that took it.

• GPQA

  #67 / 256

  Graduate-level science questions written to be hard to look up.

  85.5%

  Self-reported

• MMLU-Pro

  #16 / 142

  Broad knowledge and reasoning across 14 subjects, ten answer choices each.

  86.1%

  Self-reported

• SWE-Bench Verified

  #59 / 120

  Fixing real bugs in real GitHub projects, checked by the projects' own tests.

  72.4%

  Self-reported

• Humanity's Last Exam

  #26 / 108

  Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions

  48.5%

  Self-reported · In the text score· In the vision score

• MMMU-Pro

  #35 / 73

  College-level questions that need both an image and text to answer.

  75.0%

  Self-reported · In the vision score

• IFEval

  #1 / 70

  Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints

  95.0%

  Self-reported

• BrowseComp

  #40 / 68

  Finding hard-to-locate facts by browsing the web.

  61.0%

  Self-reported · In the agents score

• LiveCodeBench v6

  #25 / 67

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  80.7%

  Self-reported

• MMMU

  #10 / 66

  MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.

  82.3%

  Self-reported

• CharXiv-R

  #29 / 60

  Reading and reasoning about charts from scientific papers.

  79.5%

  Self-reported

• Terminal-Bench 2.0

  #46 / 54

  Getting real work done in a command-line terminal.

  41.6%

  Self-reported

• MMMLU

  #26 / 51

  Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.

  85.9%

  Self-reported

• MMLU-Redux

  #13 / 50

  An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.

  93.2%

  Self-reported

• IFBench

  #16 / 44

  Instruction Following Benchmark evaluating model's ability to follow complex instructions

  76.5%

  Self-reported

• Multi-Challenge

  #12 / 41

  MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.

  60.8%

  Self-reported

• MathVision

  #11 / 38

  MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.

  86.0%

  Self-reported

• SuperGPQA

  #10 / 37

  SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.

  65.6%

  Self-reported

• AI2D

  #7 / 35

  AI2D is a dataset of 4,903 illustrative diagrams from grade school natural sciences (such as food webs, human physiology, and life cycles) with over 15,000 multiple choice questions and answers. The benchmark evaluates diagram understanding and visual reasoning capabilities, requiring models to interpret diagrammatic elements, relationships, and structure to answer questions about scientific concepts represented in visual form.

  92.9%

  Self-reported

• HMMT 2025

  #14 / 33

  Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds

  92.0%

  Self-reported

• MMLU-ProX

  #6 / 32

  Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.

  82.2%

  Self-reported

• RealWorldQA

  #13 / 32

  RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts.

  83.7%

  Self-reported

• Include

  #7 / 31

  Include benchmark - specific documentation not found in official sources

  81.6%

  Self-reported

• LVBench

  #12 / 31

  LVBench is an extreme long video understanding benchmark designed to evaluate multimodal models on videos up to two hours in duration. It contains 6 major categories and 21 subcategories, with videos averaging five times longer than existing datasets. The benchmark addresses applications requiring comprehension of extremely long videos.

  73.6%

  Self-reported

• ERQA

  #16 / 29

  Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions

  60.5%

  Self-reported

• HMMT25

  #7 / 28

  Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds

  89.8%

  Self-reported

• OSWorld-Verified

  #25 / 28

  OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.

  56.2%

  Self-reported

• VideoMMMU

  #15 / 28

  Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.

  82.3%

  Self-reported

• MMStar

  #5 / 27

  MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans to evaluate 6 core capabilities and 18 detailed axes. The benchmark addresses issues of visual content unnecessity and unintentional data leakage in existing multimodal evaluations.

  81.0%

  Self-reported

• ScreenSpot Pro

  #11 / 27

  ScreenSpot-Pro is a novel GUI grounding benchmark designed to rigorously evaluate the grounding capabilities of multimodal large language models (MLLMs) in professional high-resolution computing environments. The benchmark comprises 1,581 instructions across 23 applications spanning 5 industries and 3 operating systems, featuring authentic high-resolution images from professional domains with expert annotations. Unlike previous benchmarks that focus on cropped screenshots in consumer applications, ScreenSpot-Pro addresses the complexity and diversity of real-world professional software scenarios, revealing significant performance gaps in current MLLM GUI perception capabilities.

  70.3%

  Self-reported

• MathVista-Mini

  #2 / 26

  MathVista-Mini is a smaller version of the MathVista benchmark that evaluates mathematical reasoning in visual contexts. It consists of examples derived from multimodal datasets involving mathematics, combining challenges from diverse mathematical and visual tasks to assess foundation models' ability to solve problems requiring both visual understanding and mathematical reasoning.

  87.8%

  Self-reported

• AA-LCR

  #13 / 25

  Agent Arena Long Context Reasoning benchmark

  66.1%

  Self-reported

• HealthBench Hard

  #15 / 24

  A challenging variation of HealthBench that evaluates large language models' performance and safety in healthcare through 5,000 multi-turn conversations with particularly rigorous evaluation criteria validated by 262 physicians from 60 countries

  22.4%

  Verified

• LongBench v2

  #9 / 24

  LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.

  60.6%

  Self-reported

• OCRBench

  #8 / 24

  OCRBench: Comprehensive evaluation benchmark for assessing Optical Character Recognition (OCR) capabilities in Large Multimodal Models across text recognition, scene text VQA, and document understanding tasks

  89.4%

  Self-reported

• WMT24++

  #10 / 24

  WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.

  77.6%

  Self-reported

• PolyMATH

  #5 / 23

  Polymath is a challenging multi-modal mathematical reasoning benchmark designed to evaluate the general cognitive reasoning abilities of Multi-modal Large Language Models (MLLMs). The benchmark comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning.

  71.2%

  Self-reported

• t2-bench

  #15 / 23

  t2-bench is a benchmark for evaluating agentic tool use capabilities, measuring how well models can select, sequence, and utilize tools to solve complex tasks. It tests autonomous planning and execution in multi-step scenarios.

  79.0%

  Self-reported

• BFCL-V4

  #6 / 21

  Berkeley Function Calling Leaderboard V4 (BFCL-V4) evaluates LLMs on their ability to accurately call functions and APIs, including simple, multiple, parallel, and nested function calls across diverse programming scenarios.

  68.5%

  Self-reported

• C-Eval

  #8 / 20

  C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. It comprises 13,948 multiple-choice questions across 52 diverse disciplines spanning humanities, science, and engineering, with four difficulty levels: middle school, high school, college, and professional. The benchmark includes C-Eval Hard, a subset of very challenging subjects requiring advanced reasoning abilities.

  90.5%

  Self-reported

• CodeForces

  #11 / 20

  A competitive programming benchmark using problems from the CodeForces platform. The benchmark evaluates code generation capabilities of LLMs on algorithmic problems with difficulty ratings ranging from 800 to 2400. Problems cover diverse algorithmic categories including dynamic programming, graph algorithms, data structures, and mathematical problems with standardized evaluation through direct platform submission.

  0.0%

  Self-reported

• Hallusion Bench

  #1 / 20

  A comprehensive benchmark designed to evaluate image-context reasoning in large visual-language models (LVLMs) by challenging models with 346 images and 1,129 carefully crafted questions to assess language hallucination and visual illusion

  70.0%

  Self-reported

• MMBench-V1.1

  #3 / 20

  Version 1.1 of MMBench, an improved bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks.

  92.6%

  Self-reported

• OmniDocBench 1.5

  #9 / 20

  OmniDocBench 1.5 is a comprehensive benchmark for evaluating multimodal large language models on document understanding tasks, including OCR, document parsing, information extraction, and visual question answering across diverse document types. Lower Overall Edit Distance scores are better.

  88.9%

  Self-reported

• CC-OCR

  #7 / 18

  A comprehensive OCR benchmark for evaluating Large Multimodal Models (LMMs) in literacy. Comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. Contains 39 subsets with 7,058 fully annotated images, 41% sourced from real applications. Tests capabilities including text grounding, multi-orientation text recognition, and detecting hallucination/repetition across diverse visual challenges.

  81.0%

  Self-reported

• MVBench

  #5 / 18

  A comprehensive multi-modal video understanding benchmark covering 20 challenging video tasks that require temporal understanding beyond single-frame analysis. Tasks span from perception to cognition, including action recognition, temporal reasoning, spatial reasoning, object interaction, scene transition, and counterfactual inference. Uses a novel static-to-dynamic method to systematically generate video tasks from existing annotations.

  74.6%

  Self-reported

• SimpleVQA

  #16 / 17

  SimpleVQA is a visual question answering benchmark focused on simple queries.

  0.6%

  Self-reported

• BabyVision

  #13 / 16

  A benchmark for early-stage visual reasoning and perception on child-like vision tasks.

  44.6%

  Self-reported

• ODinW

  #14 / 16

  Object Detection in the Wild (ODinW) benchmark for evaluating object detection models' task-level transfer ability across diverse real-world datasets in terms of prediction accuracy and adaptation efficiency

  41.1%

  Self-reported

• BrowseComp-zh

  #10 / 15

  A high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web, consisting of 289 multi-hop questions spanning 11 diverse domains including Film & TV, Technology, Medicine, and History. Questions are reverse-engineered from short, objective, and easily verifiable answers, requiring sophisticated reasoning and information reconciliation beyond basic retrieval. The benchmark addresses linguistic, infrastructural, and censorship-related complexities in Chinese web environments.

  62.1%

  Self-reported

• Global PIQA

  #9 / 15

  Global PIQA is a multilingual commonsense reasoning benchmark that evaluates physical interaction knowledge across 100 languages and cultures. It tests AI systems' understanding of physical world knowledge in diverse cultural contexts through multiple choice questions about everyday situations requiring physical commonsense.

  87.5%

  Self-reported

• ZEROBench

  #11 / 15

  ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks.

  0.1%

  Self-reported

• WideSearch

  #10 / 14

  WideSearch is an agentic search benchmark that evaluates models' ability to perform broad, parallel search operations across multiple sources. It tests wide-coverage information retrieval and synthesis capabilities.

  61.1%

  Self-reported

• HealthBench (hard)

  #7 / 13

  22.4%

  Unverified

• MAXIFE

  #5 / 12

  MAXIFE is a multilingual benchmark evaluating LLMs on instruction following and execution across multiple languages and cultural contexts.

  88.0%

  Self-reported

• MedXpertQA

  #3 / 12

  A comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning, featuring 4,460 questions spanning 17 specialties and 11 body systems. Includes both text-only and multimodal subsets with expert-level exam questions incorporating diverse medical images and rich clinical information.

  62.4%

  Self-reported

• NOVA-63

  #5 / 12

  NOVA-63 is a multilingual evaluation benchmark covering 63 languages, designed to assess LLM performance across diverse linguistic contexts and tasks.

  58.1%

  Self-reported

• VideoMME w sub.

  #5 / 12

  The first-ever comprehensive evaluation benchmark of Multi-modal LLMs in Video analysis. Features 900 videos (254 hours) with 2,700 question-answer pairs covering 6 primary visual domains and 30 subfields. Evaluates temporal understanding across short (11 seconds) to long (1 hour) videos with multi-modal inputs including video frames, subtitles, and audio.

  87.0%

  Self-reported

• VITA-Bench

  #5 / 11

  VITA-Bench evaluates AI agents on real-world virtual task automation, measuring their ability to complete complex multi-step tasks in simulated environments.

  41.9%

  Self-reported

• DeepPlanning

  #8 / 10

  DeepPlanning evaluates LLMs on complex multi-step planning tasks requiring long-horizon reasoning, goal decomposition, and strategic decision-making.

  22.6%

  Self-reported

• MLVU

  #6 / 10

  A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization.

  85.9%

  Self-reported

• VideoMME w/o sub.

  #2 / 10

  Video-MME is a comprehensive evaluation benchmark for multi-modal large language models in video analysis. It features 900 videos across 6 primary visual domains with 30 subfields, ranging from 11 seconds to 1 hour in duration, with 2,700 question-answer pairs. The benchmark evaluates MLLMs' capabilities in processing sequential visual data and multi-modal content including video frames, subtitles, and audio.

  82.8%

  Self-reported

• CountBench

  #2 / 9

  CountBench evaluates object counting capabilities in visual understanding.

  1.0%

  Self-reported

• DynaMath

  #2 / 9

  A multimodal mathematics and reasoning benchmark focused on dynamic visual problem solving.

  87.7%

  Self-reported

• OJBench

  #3 / 9

  OJBench is a competition-level code benchmark designed to assess the competitive-level code reasoning abilities of large language models. It comprises 232 programming competition problems from NOI and ICPC, categorized into Easy, Medium, and Hard difficulty levels. The benchmark evaluates models' ability to solve complex competitive programming challenges using Python and C++.

  40.1%

  Self-reported

• RefCOCO-avg

  #6 / 9

  RefCOCO-avg measures object grounding accuracy averaged across RefCOCO, RefCOCO+, and RefCOCOg benchmarks.

  0.9%

  Self-reported

• AndroidWorld_SR

  #3 / 8

  AndroidWorld Success Rate (SR) benchmark - A dynamic benchmarking environment for autonomous agents operating on Android devices. Evaluates agents on 116 programmatic tasks across 20 real-world Android apps using multimodal inputs (screen screenshots, accessibility trees, and natural language instructions). Measures success rate of agents completing tasks like sending messages, creating calendar events, and navigating mobile interfaces. Published at ICLR 2025. Best current performance: 30.6% success rate (M3A agent) vs 80.0% human performance.

  64.2%

  Self-reported

• EmbSpatialBench

  #2 / 8

  EmbSpatialBench evaluates embodied spatial understanding and reasoning capabilities.

  0.8%

  Self-reported

• RefSpatialBench

  #5 / 8

  RefSpatialBench evaluates spatial reference understanding and grounding.

  0.7%

  Self-reported

• MMLongBench-Doc

  #3 / 7

  MMLongBench-Doc evaluates long document understanding capabilities in vision-language models.

  0.6%

  Self-reported

• MMVU

  #4 / 7

  MMVU (Multimodal Multi-disciplinary Video Understanding) is a benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, testing comprehension and reasoning capabilities on video content.

  73.3%

  Self-reported

• Seal-0

  #3 / 7

  Seal-0 is a benchmark for evaluating agentic search capabilities, testing models' ability to navigate and retrieve information using tools.

  47.2%

  Self-reported

• V*

  #4 / 7

  A visual reasoning benchmark evaluating multimodal inference under challenging spatial and grounded tasks.

  93.7%

  Self-reported

• VLMsAreBlind

  #3 / 6

  A vision-language benchmark that probes blind spots and brittle reasoning in multimodal models.

  96.9%

  Self-reported

• ZEROBench-Sub

  #1 / 6

  ZEROBench-Sub is a subset of the ZEROBench benchmark.

  0.4%

  Self-reported

• Hypersim

  #2 / 4

  Hypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes.

  0.1%

  Self-reported

• LingoQA

  #2 / 4

  A benchmark for multimodal spatial-language understanding and visual-linguistic question answering.

  82.0%

  Self-reported

• SUNRGBD

  #2 / 4

  SUNRGBD evaluates RGB-D scene understanding and 3D grounding capabilities.

  0.4%

  Self-reported

• SlakeVQA

  #2 / 4

  A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Contains 642 radiology images (CT scans, MRI scans, X-rays) covering five body parts and 14,028 bilingual English-Chinese question-answer pairs annotated by experienced physicians. Features comprehensive semantic labels and a structural medical knowledge base with both vision-only and knowledge-based questions requiring external medical knowledge reasoning.

  80.0%

  Self-reported

• TIR-Bench

  #2 / 4

  A tool-calling and multimodal interaction benchmark for testing visual instruction following and execution reliability.

  59.8%

  Self-reported

• FullStackBench en

  #2 / 3

  English subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.

  60.1%

  Self-reported

• FullStackBench zh

  #2 / 3

  Chinese subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.

  57.4%

  Self-reported

• Nuscene

  #2 / 3

  A multimodal benchmark for scene understanding and reasoning over the nuScenes autonomous driving domain.

  15.2%

  Self-reported

• PMC-VQA

  #2 / 3

  A medical visual question answering benchmark built on biomedical literature and medical figures.

  62.4%

  Self-reported