LongCat-Flash-Chat

Leaderboards

Where LongCat-Flash-Chat sits among the top 20 in each category.

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Test results

Every test LongCat-Flash-Chat has a published result on, out of 100%, with its rank among the models that took it.

• GPQA

  #135 / 256

  Graduate-level science questions written to be hard to look up.

  73.2%

  Self-reported

• MMLU-Pro

  #40 / 142

  Broad knowledge and reasoning across 14 subjects, ten answer choices each.

  82.7%

  Self-reported

• AIME 2025

  #105 / 123

  Competition maths problems from the 2025 American Invitational Mathematics Exam.

  61.3%

  Self-reported

• SWE-Bench Verified

  #89 / 120

  Fixing real bugs in real GitHub projects, checked by the projects' own tests.

  60.4%

  Self-reported

• MMLU

  #13 / 103

  Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains

  89.7%

  Self-reported

• LiveCodeBench

  #51 / 76

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  48.0%

  Self-reported

• IFEval

  #21 / 70

  Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints

  89.6%

  Self-reported

• HumanEval

  #19 / 66

  A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

  88.4%

  Self-reported

• Tau2 Telecom

  #24 / 36

  τ²-Bench telecom domain evaluates conversational agents in a dual-control environment modeled as a Dec-POMDP, where both agent and user use tools in shared telecommunications troubleshooting scenarios that test coordination and communication capabilities.

  73.7%

  Self-reported

• MATH-500

  #14 / 34

  MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

  96.4%

  Self-reported

• DROP

  #16 / 30

  DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.

  79.1%

  Self-reported

• Tau2 Retail

  #20 / 27

  τ²-bench retail domain evaluates conversational AI agents in customer service scenarios within a dual-control environment where both agent and user can interact with tools. Tests tool-agent-user interaction, rule adherence, and task consistency in retail customer support contexts.

  71.3%

  Self-reported

• Tau2 Airline

  #14 / 25

  TAU2 airline domain benchmark for evaluating conversational agents in dual-control environments where both AI agents and users interact with tools in airline customer service scenarios. Tests agent coordination, communication, and ability to guide user actions in tasks like flight booking, modifications, cancellations, and refunds.

  58.0%

  Self-reported

• Terminal-Bench

  #9 / 25

  Terminal-Bench is a benchmark for testing AI agents in real terminal environments. It evaluates how well agents can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities. The benchmark consists of a dataset of ~100 hand-crafted, human-verified tasks and an execution harness that connects language models to a terminal sandbox.

  39.5%

  Self-reported

• ZebraLogic

  #4 / 9

  ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.

  89.3%

  Self-reported

• CMMLU

  #3 / 6

  CMMLU (Chinese Massive Multitask Language Understanding) is a comprehensive Chinese benchmark that evaluates the knowledge and reasoning capabilities of large language models across 67 different subject topics. The benchmark covers natural sciences, social sciences, engineering, and humanities with multiple-choice questions ranging from basic to advanced professional levels.

  84.3%

  Self-reported