Skip to content

Anthropic Proprietary Released Jun 21, 2024 Updated

Claude 3.5 Sonnet

ToolsGuide Scores, rankings, test results and price.

Model summary

  • Text #229 / 416

    210 / 1,000

    ToolsGuide Score· votes 215, tests 205

  • Vision #109 / 167

    268 / 1,000

    ToolsGuide Score· votes only

  • Price per 1M tokens

    No listed price

Summary

Claude 3.5 Sonnet is Anthropic's model, released Jun 21, 2024. It's ranked in 2 categories and does best at text: #229 of 416, with a ToolsGuide Score of 210 against a median of 235.

It also ranks #109 of 167 for vision (268).

Its vision score uses votes only, because it has published fewer than two of the tests used for that category.

It has published results on 9 tests and has the top score on 1 of them, including BIG-Bench Hard.

Specifications

Creator
Anthropic
Licence
Proprietary
Released
Jun 21, 2024
Context window
–
Input price
–
Output price
–
Tests reported
9
Settings ranked
claude-3-5-sonnet-20240620, Claude 3.5 Sonnet (20241022), Claude 3.5 Sonnet (20240620)

Model comparison

Against the ten best text models.

Text score

ToolsGuide Score out of 1,000 · Higher is better

Vision score

ToolsGuide Score out of 1,000 · Higher is better

Input price

USD per 1M input tokens · Lower is better

Leaderboards

Where Claude 3.5 Sonnet sits among the top 20 in each category.

Text

#229 of 416 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full text leaderboard →

Vision

#109 of 167 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full vision leaderboard →

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Against its nearest rivals

Text prompts by kind · ToolsGuide Score from votes · hover for values

Text

claude-3-5-sonnet-20240620 · hover a bar for its rank

Vision

Claude 3.5 Sonnet (20241022) · hover a bar for its rank

Test results

Every test Claude 3.5 Sonnet has a published result on, out of 100%, with its rank among the models that took it.

  • GPQA

    #180 / 256

    Graduate-level science questions written to be hard to look up.

    59.4%

    Self-reported

  • MMLU-Pro

    #76 / 142

    Broad knowledge and reasoning across 14 subjects, ten answer choices each.

    76.1%

    Self-reported

  • MMLU

    #8 / 103

    Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains

    90.4%

    Self-reported

  • MATH

    #36 / 71

    MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

    71.1%

    Self-reported

  • HumanEval

    #9 / 66

    A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

    92.0%

    Self-reported

  • GSM8k

    #6 / 48

    Grade School Math 8K, a dataset of 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.

    96.4%

    Self-reported

  • MGSM

    #3 / 31

    MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.

    91.6%

    Self-reported

  • DROP

    #2 / 30

    DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.

    87.1%

    Self-reported

  • BIG-Bench Hard

    #1 / 21

    BIG-Bench Hard (BBH) is a subset of 23 challenging BIG-Bench tasks selected because prior language model evaluations did not outperform average human-rater performance. The benchmark contains 6,511 evaluation examples testing various forms of multi-step reasoning including arithmetic, logical reasoning (Boolean expressions, logical deduction), geometric reasoning, temporal reasoning, and language understanding. Tasks require capabilities such as causal judgment, object counting, navigation, pattern recognition, and complex problem solving.

    93.1%

    Self-reported