Skip to content

Mistral apache_2_0 Released Mar 17, 2025 Updated

Mistral Small 3.1 24B Instruct

ToolsGuide Scores, rankings, test results and price.

Model summary

  • Text #280 / 416

    173 / 1,000

    ToolsGuide Score· votes 181, tests 165

  • Vision #127 / 167

    233 / 1,000

    ToolsGuide Score· votes only

  • Price per 1M tokens

    No listed price

Summary

Mistral Small 3.1 24B Instruct is Mistral's model, released Mar 17, 2025. It's ranked in 2 categories and does best at text: #280 of 416, with a ToolsGuide Score of 173 against a median of 235.

It also ranks #127 of 167 for vision (233).

Its vision score uses votes only, because it has published fewer than two of the tests used for that category.

It has published results on 9 tests.

Specifications

Creator
Mistral
Licence
apache_2_0
Released
Mar 17, 2025
Context window
–
Input price
–
Output price
–
Tests reported
9
Settings ranked
mistral-small-3.1-24b-instruct-2503, Mistral Small 3.1 24B Instruct (2503)

Model comparison

Against the ten best text models.

Text score

ToolsGuide Score out of 1,000 · Higher is better

Vision score

ToolsGuide Score out of 1,000 · Higher is better

Input price

USD per 1M input tokens · Lower is better

Leaderboards

Where Mistral Small 3.1 24B Instruct sits among the top 20 in each category.

Text

#280 of 416 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full text leaderboard →

Vision

#127 of 167 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full vision leaderboard →

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Against its nearest rivals

Text prompts by kind · ToolsGuide Score from votes · hover for values

Text

mistral-small-3.1-24b-instruct-2503 · hover a bar for its rank

Vision

Mistral Small 3.1 24B Instruct (2503) · hover a bar for its rank

Test results

Every test Mistral Small 3.1 24B Instruct has a published result on, out of 100%, with its rank among the models that took it.

  • GPQA

    #212 / 256

    Graduate-level science questions written to be hard to look up.

    46.0%

    Self-reported

  • MMLU-Pro

    #107 / 142

    Broad knowledge and reasoning across 14 subjects, ten answer choices each.

    66.8%

    Self-reported

  • MMLU

    #59 / 103

    Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains

    80.6%

    Self-reported

  • MATH

    #42 / 71

    MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

    69.3%

    Self-reported

  • HumanEval

    #19 / 66

    A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

    88.4%

    Self-reported

  • MMMU

    #49 / 66

    MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.

    59.3%

    Self-reported

  • SimpleQA

    #41 / 47

    SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.

    10.4%

    Self-reported

  • MBPP

    #18 / 33

    MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality.

    0.7%

    Self-reported

  • TriviaQA

    #4 / 18

    A large-scale reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents (six per question on average) that provide high quality distant supervision for answering the questions. The dataset features relatively complex, compositional questions with considerable syntactic and lexical variability, requiring cross-sentence reasoning to find answers.

    80.5%

    Self-reported