Skip to content

xAI Proprietary Released Aug 13, 2024 Updated

Grok-2

ToolsGuide Scores, rankings, test results and price.

Model summary

  • Text #232 / 419

    219 / 1,000

    ToolsGuide Score· votes 208, tests 248

  • Price per 1M tokens

    No listed price

Summary

Grok-2 is xAI's model, released Aug 13, 2024. It's ranked in 1 category and does best at text: #232 of 419, with a ToolsGuide Score of 219 against a median of 252.

It has published results on 8 tests.

Specifications

Creator
xAI
Licence
Proprietary
Released
Aug 13, 2024
Context window
–
Input price
–
Output price
–
Tests reported
8
Settings ranked
Grok 2 (2024-08-13)

Model comparison

Against the ten best text models.

Text score

ToolsGuide Score out of 1,000 · Higher is better

Input price

USD per 1M input tokens · Lower is better

Leaderboards

Where Grok-2 sits among the top 20 in each category.

Text

#232 of 419 · ToolsGuide Score out of 1,000, where 1,000 is a flawless model

Full text leaderboard →

By task

Scored from votes on each kind of prompt alone, on the same 1,000-point scale.

Against its nearest rivals

Text prompts by kind · ToolsGuide Score from votes · hover for values

Text

Grok 2 (2024-08-13) · hover a bar for its rank

Test results

Every test Grok-2 has a published result on, out of 100%, with its rank among the models that took it.

  • GPQA

    #195 / 265

    Graduate-level science questions written to be hard to look up.

    56.0%

    Self-reported

  • MMLU-Pro

    #80 / 142

    Broad knowledge and reasoning across 14 subjects, ten answer choices each.

    75.5%

    Self-reported

  • MMLU

    #24 / 104

    Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains

    87.5%

    Self-reported

  • MATH

    #26 / 71

    MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.

    76.1%

    Self-reported

  • HumanEval

    #21 / 66

    A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

    88.4%

    Self-reported

  • MMMU

    #40 / 66

    MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.

    66.1%

    Self-reported

  • MathVista

    #17 / 40

    MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.

    69.0%

    Self-reported

  • DocVQA

    #9 / 28

    A dataset for Visual Question Answering on document images containing 50,000 questions defined on 12,000+ document images. The benchmark tests AI's ability to understand document structure and content, requiring models to comprehend document layout and perform information retrieval to answer questions about document images.

    93.6%

    Self-reported