Qwen3.8-27B

Leaderboards

Where Qwen3.8-27B sits among the top 20 in each category.

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Test results

Every test Qwen3.8-27B has a published result on, out of 100%, with its rank among the models that took it.

• GPQA

  #34 / 256

  Graduate-level science questions written to be hard to look up.

  89.2%

  Self-reported

• Humanity's Last Exam

  #54 / 108

  Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions

  30.8%

  Self-reported · In the text score· In the vision score

• LiveCodeBench v6

  #5 / 67

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  90.3%

  Self-reported

• SWE-Bench Pro

  #18 / 61

  Harder, longer software engineering tasks from real codebases.

  61.7%

  Self-reported · In the coding score

• CharXiv-R

  #6 / 60

  Reading and reasoning about charts from scientific papers.

  90.2%

  Self-reported

• DeepSWE 1.1

  #45 / 50

  DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness.

  42.2%

  Self-reported · In the coding score

• Terminal-Bench 2.1

  #26 / 46

  Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal. It evaluates how well models handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, data science workflows, and security tasks.

  73.0%

  Self-reported

• IFBench

  #9 / 44

  Instruction Following Benchmark evaluating model's ability to follow complex instructions

  79.5%

  Self-reported

• MathVision

  #4 / 38

  MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.

  94.6%

  Self-reported

• RealWorldQA

  #7 / 32

  RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts.

  85.9%

  Self-reported

• ERQA

  #10 / 29

  Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions

  65.5%

  Self-reported

• OSWorld-Verified

  #3 / 28

  OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.

  84.3%

  Self-reported

• NL2Repo

  #19 / 25

  NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.

  42.3%

  Self-reported · In the coding score

• Agents' Last Exam

  #10 / 24

  Agents' Last Exam is a challenging benchmark for AI agents on hard, long-horizon tasks that test sustained reasoning, planning, and tool use, reported with and without tool access.

  42.9%

  Self-reported · In the agents score

• OmniDocBench 1.5

  #4 / 20

  OmniDocBench 1.5 is a comprehensive benchmark for evaluating multimodal large language models on document understanding tasks, including OCR, document parsing, information extraction, and visual question answering across diverse document types. Lower Overall Edit Distance scores are better.

  91.1%

  Self-reported

• BabyVision

  #5 / 16

  A benchmark for early-stage visual reasoning and perception on child-like vision tasks.

  85.6%

  Self-reported

• Job Bench

  #9 / 9

  Job Bench evaluates AI agents on realistic professional tasks that require multi-step planning, research, and production of work artifacts.

  33.4%

  Self-reported

• AndroidWorld

  #4 / 8

  AndroidWorld evaluates an agent's ability to operate in real Android GUI environments, completing multi-step tasks by perceiving screen content and executing touch/type actions.

  81.9%

  Self-reported

• SWE-Bench Multimodal

  #4 / 7

  SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding.

  38.6%

  Self-reported

• ClawEval-MM

  #3 / 6

  ClawEval-MM is the multimodal variant of ClawEval, evaluating agentic problem solving with visual inputs.

  56.9%

  Self-reported

• CoWorkBench

  #4 / 6

  CoWorkBench is Qwen's internal cowork benchmark for evaluating long-horizon office and productivity agent tasks across domains such as computer science, finance, law, and medicine.

  70.7%

  Self-reported

• Vision2Web

  #4 / 5

  Vision2Web evaluates multimodal models on converting visual designs and screenshots into functional web pages, measuring end-to-end design-to-code capability.

  62.9%

  Self-reported

• RecreationBench

  #3 / 3

  RecreationBench is Qwen's long-horizon application-recreation benchmark for hybrid agents across Ubuntu, macOS, Windows, Android, and the web.

  47.1%

  Self-reported

• QwenSWEBench

  #2 / 2

  QwenSWEBench is Qwen's software-engineering agent benchmark for repository-level issue resolution.

  79.0%

  Unverified

• SWE-MM

  #1 / 1

  SWE-MM evaluates software-engineering agents on repository tasks that require understanding both source code and visual evidence.

  38.6%

  Unverified

• WebArena-Verified

  #1 / 1

  WebArena-Verified evaluates browser agents on realistic web tasks using a verified task set and execution-based grading.

  64.8%

  Unverified