Qwen3.5-397B-A17B

Leaderboards

Where Qwen3.5-397B-A17B sits among the top 20 in each category.

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Test results

Every test Qwen3.5-397B-A17B has a published result on, out of 100%, with its rank among the models that took it.

• GPQA

  #39 / 256

  Graduate-level science questions written to be hard to look up.

  88.4%

  Self-reported

• MMLU-Pro

  #6 / 142

  Broad knowledge and reasoning across 14 subjects, ten answer choices each.

  87.8%

  Self-reported

• SWE-Bench Verified

  #33 / 120

  Fixing real bugs in real GitHub projects, checked by the projects' own tests.

  76.4%

  Self-reported

• Humanity's Last Exam

  #57 / 108

  Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions

  28.7%

  Self-reported · In the text score· In the vision score

• IFEval

  #9 / 70

  Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints

  92.6%

  Self-reported

• BrowseComp

  #33 / 68

  Finding hard-to-locate facts by browsing the web.

  69.0%

  Self-reported · In the agents score

• LiveCodeBench v6

  #18 / 67

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  83.6%

  Self-reported

• Terminal-Bench 2.0

  #33 / 54

  Getting real work done in a command-line terminal.

  52.5%

  Self-reported

• MMMLU

  #17 / 51

  Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.

  88.5%

  Self-reported

• MMLU-Redux

  #2 / 50

  An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.

  94.9%

  Self-reported

• SWE-bench Multilingual

  #29 / 49

  A multilingual benchmark for issue resolving in software engineering that covers Java, TypeScript, JavaScript, Go, Rust, C, and C++. Contains 1,632 high-quality instances carefully annotated from 2,456 candidates by 68 expert annotators, designed to evaluate Large Language Models across diverse software ecosystems beyond Python.

  69.3%

  Self-reported

• IFBench

  #16 / 44

  Instruction Following Benchmark evaluating model's ability to follow complex instructions

  76.5%

  Self-reported

• Toolathlon

  #36 / 42

  Tool Decathlon is a comprehensive benchmark for evaluating AI agents' ability to use multiple tools across diverse task categories. It measures proficiency in tool selection, sequencing, and execution across ten different tool-use scenarios.

  38.3%

  Self-reported · In the agents score

• Multi-Challenge

  #5 / 41

  MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.

  67.6%

  Self-reported

• SuperGPQA

  #5 / 37

  SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.

  70.4%

  Self-reported

• HMMT 2025

  #7 / 33

  Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds

  94.8%

  Self-reported

• MMLU-ProX

  #3 / 32

  Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.

  84.7%

  Self-reported

• Include

  #3 / 31

  Include benchmark - specific documentation not found in official sources

  85.6%

  Self-reported

• HMMT25

  #3 / 28

  Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds

  92.7%

  Self-reported

• AIME 2026

  #18 / 26

  All 30 problems from the 2026 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.

  91.3%

  Self-reported

• AA-LCR

  #8 / 25

  Agent Arena Long Context Reasoning benchmark

  68.7%

  Self-reported

• LongBench v2

  #2 / 24

  LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.

  63.2%

  Self-reported

• WMT24++

  #8 / 24

  WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.

  78.9%

  Self-reported

• PolyMATH

  #4 / 23

  Polymath is a challenging multi-modal mathematical reasoning benchmark designed to evaluate the general cognitive reasoning abilities of Multi-modal Large Language Models (MLLMs). The benchmark comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning.

  73.3%

  Self-reported

• t2-bench

  #4 / 23

  t2-bench is a benchmark for evaluating agentic tool use capabilities, measuring how well models can select, sequence, and utilize tools to solve complex tasks. It tests autonomous planning and execution in multi-step scenarios.

  86.7%

  Self-reported

• IMO-AnswerBench

  #17 / 22

  IMO-AnswerBench is a benchmark for evaluating mathematical reasoning capabilities on International Mathematical Olympiad (IMO) problems, focusing on answer generation and verification.

  80.9%

  Self-reported

• BFCL-V4

  #3 / 21

  Berkeley Function Calling Leaderboard V4 (BFCL-V4) evaluates LLMs on their ability to accurately call functions and APIs, including simple, multiple, parallel, and nested function calls across diverse programming scenarios.

  72.9%

  Self-reported

• C-Eval

  #3 / 20

  C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. It comprises 13,948 multiple-choice questions across 52 diverse disciplines spanning humanities, science, and engineering, with four difficulty levels: middle school, high school, college, and professional. The benchmark includes C-Eval Hard, a subset of very challenging subjects requiring advanced reasoning abilities.

  93.0%

  Self-reported

• BrowseComp-zh

  #2 / 15

  A high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web, consisting of 289 multi-hop questions spanning 11 diverse domains including Film & TV, Technology, Medicine, and History. Questions are reverse-engineered from short, objective, and easily verifiable answers, requiring sophisticated reasoning and information reconciliation beyond basic retrieval. The benchmark addresses linguistic, infrastructural, and censorship-related complexities in Chinese web environments.

  70.3%

  Self-reported

• Global PIQA

  #5 / 15

  Global PIQA is a multilingual commonsense reasoning benchmark that evaluates physical interaction knowledge across 100 languages and cultures. It tests AI systems' understanding of physical world knowledge in diverse cultural contexts through multiple choice questions about everyday situations requiring physical commonsense.

  89.8%

  Self-reported

• WideSearch

  #7 / 14

  WideSearch is an agentic search benchmark that evaluates models' ability to perform broad, parallel search operations across multiple sources. It tests wide-coverage information retrieval and synthesis capabilities.

  74.0%

  Self-reported

• MAXIFE

  #3 / 12

  MAXIFE is a multilingual benchmark evaluating LLMs on instruction following and execution across multiple languages and cultural contexts.

  88.2%

  Self-reported

• NOVA-63

  #1 / 12

  NOVA-63 is a multilingual evaluation benchmark covering 63 languages, designed to assess LLM performance across diverse linguistic contexts and tasks.

  59.1%

  Self-reported

• VITA-Bench

  #1 / 11

  VITA-Bench evaluates AI agents on real-world virtual task automation, measuring their ability to complete complex multi-step tasks in simulated environments.

  49.7%

  Self-reported

• DeepPlanning

  #3 / 10

  DeepPlanning evaluates LLMs on complex multi-step planning tasks requiring long-horizon reasoning, goal decomposition, and strategic decision-making.

  34.3%

  Self-reported

• MCP-Mark

  #6 / 9

  MCP-Mark evaluates LLMs on their ability to use Model Context Protocol (MCP) tools effectively, testing tool discovery, selection, invocation, and result interpretation across diverse MCP server scenarios.

  46.1%

  Self-reported

• Seal-0

  #4 / 7

  Seal-0 is a benchmark for evaluating agentic search capabilities, testing models' ability to navigate and retrieve information using tools.

  46.9%

  Self-reported

• SecCodeBench

  #1 / 2

  SecCodeBench evaluates LLM coding agents on secure code generation and vulnerability detection, testing the ability to produce code that is both functional and free from security vulnerabilities.

  68.3%

  Unverified