Skip to content

Anthropic Proprietary Released Oct 22, 2024 Updated

Claude 3.5 Haiku

ToolsGuide Scores, rankings, test results and price.

Model summary

  • Text #275 / 416

    181 / 1,000

    ToolsGuide Score· votes 198, tests 167

  • Vision #126 / 167

    233 / 1,000

    ToolsGuide Score· votes only

  • Price per 1M tokens

    No listed price

Summary

Claude 3.5 Haiku is Anthropic's model, released Oct 22, 2024. It's ranked in 2 categories and does best at text: #275 of 416, with a ToolsGuide Score of 181 against a median of 235.

It also ranks #126 of 167 for vision (233).

Its vision score uses votes only, because it has published fewer than two of the tests used for that category.

It has published results on 9 tests.

Specifications

Creator
Anthropic
Licence
Proprietary
Released
Oct 22, 2024
Context window
–
Input price
–
Output price
–
Tests reported
9
Settings ranked
claude-3-5-haiku-20241022, Claude 3.5 Haiku (20241022)

Model comparison

Against the ten best text models.

Text score

ToolsGuide Score out of 1,000 · Higher is better

Vision score

ToolsGuide Score out of 1,000 · Higher is better

Input price

USD per 1M input tokens · Lower is better

Leaderboards

Where Claude 3.5 Haiku sits among the top 20 in each category.

Text

#275 of 416 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full text leaderboard →

Vision

#126 of 167 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full vision leaderboard →

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Against its nearest rivals

Text prompts by kind · ToolsGuide Score from votes · hover for values

Text

claude-3-5-haiku-20241022 · hover a bar for its rank

Vision

Claude 3.5 Haiku (20241022) · hover a bar for its rank

Test results

Every test Claude 3.5 Haiku has a published result on, out of 100%, with its rank among the models that took it.

  • GPQA

    #224 / 256

    Graduate-level science questions written to be hard to look up.

    41.6%

    Self-reported

  • MMLU-Pro

    #111 / 142

    Broad knowledge and reasoning across 14 subjects, ten answer choices each.

    65.0%

    Self-reported

  • SWE-Bench Verified

    #111 / 120

    Fixing real bugs in real GitHub projects, checked by the projects' own tests.

    40.6%

    Self-reported

  • MATH

    #41 / 71

    MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

    69.4%

    Self-reported

  • HumanEval

    #25 / 66

    A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

    88.1%

    Self-reported

  • MGSM

    #15 / 31

    MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.

    85.6%

    Self-reported

  • DROP

    #9 / 30

    DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.

    83.1%

    Self-reported

  • TAU-bench Retail

    #24 / 25

    A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.

    51.0%

    Self-reported

  • TAU-bench Airline

    #22 / 23

    Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.

    22.8%

    Self-reported