Nemotron 3 Ultra (550B A55B)

Leaderboards

Where Nemotron 3 Ultra (550B A55B) sits among the top 20 in each category.

Price and rivals

How Nemotron 3 Ultra (550B A55B) compares with the models around it.

Test results

Every test Nemotron 3 Ultra (550B A55B) has a published result on, out of 100%, with its rank among the models that took it.

• GPQA

  #54 / 256

  Graduate-level science questions written to be hard to look up.

  87.0%

  Self-reported

• MMLU-Pro

  #10 / 142

  Broad knowledge and reasoning across 14 subjects, ten answer choices each.

  86.8%

  Self-reported

• SWE-Bench Verified

  #67 / 120

  Fixing real bugs in real GitHub projects, checked by the projects' own tests.

  70.7%

  Self-reported

• Humanity's Last Exam

  #43 / 108

  Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions

  37.4%

  Self-reported · In the text score· In the vision score

• BrowseComp

  #58 / 68

  Finding hard-to-locate facts by browsing the web.

  44.4%

  Self-reported · In the agents score

• LiveCodeBench v6

  #8 / 67

  LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.

  89.0%

  Self-reported

• SWE-bench Multilingual

  #31 / 49

  A multilingual benchmark for issue resolving in software engineering that covers Java, TypeScript, JavaScript, Go, Rust, C, and C++. Contains 1,632 high-quality instances carefully annotated from 2,456 candidates by 68 expert annotators, designed to evaluate Large Language Models across diverse software ecosystems beyond Python.

  67.7%

  Self-reported

• Terminal-Bench 2.1

  #39 / 46

  Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal. It evaluates how well models handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, data science workflows, and security tasks.

  56.4%

  Self-reported

• IFBench

  #3 / 44

  Instruction Following Benchmark evaluating model's ability to follow complex instructions

  81.7%

  Self-reported

• Multi-Challenge

  #7 / 41

  MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.

  63.8%

  Self-reported

• MMLU-ProX

  #5 / 32

  Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.

  83.0%

  Self-reported

• SciCode

  #14 / 30

  SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills.

  44.6%

  Self-reported

• Finance Agent v2

  #22 / 28

  Finance Agent v2 is an agentic financial-analysis benchmark from Vals that evaluates models on real-world finance workflows, measuring their ability to retrieve and reason over financial documents, perform multi-step calculations, and produce accurate analyses.

  37.5%

  Self-reported

• AA-LCR

  #14 / 25

  Agent Arena Long Context Reasoning benchmark

  65.4%

  Self-reported

• LongBench v2

  #4 / 24

  LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.

  61.9%

  Self-reported

• WMT24++

  #6 / 24

  WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.

  83.7%

  Self-reported

• IMO-AnswerBench

  #1 / 22

  IMO-AnswerBench is a benchmark for evaluating mathematical reasoning capabilities on International Mathematical Olympiad (IMO) problems, focusing on answer generation and verification.

  92.3%

  Self-reported

• Finance Agent

  #8 / 8

  Finance Agent is a benchmark for evaluating AI models on agentic financial analysis tasks, testing their ability to process financial data, perform calculations, and generate accurate analyses across various financial domains.

  53.7%

  Self-reported

• TAU3-Bench

  #8 / 8

  TAU3-Bench is a benchmark for evaluating general-purpose agent capabilities, testing models on multi-turn interactions with simulated user models, retrieval, and complex decision-making scenarios.

  22.6%

  Self-reported

• PinchBench

  #1 / 7

  PinchBench evaluates coding agents on real-world agentic coding tasks, measuring both best-case and average performance across complex software engineering scenarios.

  90.0%

  Self-reported

• CritPT

  #6 / 6

  CritPT is a challenging reasoning benchmark reported by Qwen for evaluating frontier mathematical and critical problem-solving capability.

  3.1%

  Self-reported

• OmniScience

  #1 / 6

  OmniScience is a broad scientific knowledge and reasoning benchmark that measures both answer accuracy and non-hallucination (calibrated abstention) across science domains.

  78.7%

  Self-reported

• RULER

  #1 / 5

  RULER v1 is a synthetic long-context benchmark for measuring how model quality degrades as input length increases. This packaging follows the public standalone NVIDIA RULER implementation with 13 official tasks spanning retrieval, multi-hop tracing, aggregation, and QA.

  94.7%

  Self-reported

• GDPval

  #3 / 4

  GDPval is an OpenAI benchmark evaluating AI models on economically valuable, real-world knowledge-work tasks spanning many professional occupations and industries.

  46.7%

  Self-reported

• ProfBench

  #1 / 4

  ProfBench evaluates models on professional-domain reasoning and knowledge-work tasks, including search-augmented question answering across expert fields.

  56.0%

  Self-reported

• Apex

  #1 / 2

  Apex is a challenging frontier reasoning benchmark testing advanced multi-step problem solving across difficult STEM and logical tasks.

  84.8%

  Unverified