Benchmark · Version 1

Quiz Generation

This benchmark evaluates how AI tools turn supplied study material into quizzes and practice tests, then grade, explain, and track what learners actually understood.

What this benchmark is

This benchmark is for AI products that work from supplied study material. It asks whether a tool can turn that material into quizzes and practice tests, then deliver them, grade responses, explain mistakes, and track understanding.

Buyers can use it to compare products for classroom and learner use. The benchmark shows whether the tool handles assessment content, answer evaluation, learner feedback, ongoing practice, and reporting in ways that stay faithful to the source material.

Version 1 uses the approved Electric Circuits textbook corpus. The registered resource Quiz generation fixture material — Foundation Science 8 chapters supplies those chapter materials for this benchmark.

In scope

  • Turning supplied study material into quizzes and practice tests.
  • Delivering quizzes to individual learners or classes.
  • Grading fixed-answer, free-form, and partly correct responses.
  • Giving feedback that explains what is correct, missing, or wrong.
  • Adapting later practice to strengths and weaknesses, including across sessions.
  • Showing learner progress over time.
  • Showing class reports about what the class struggled with.

Out of scope

  • Product attributes, which are captured separately and never scored.
7Capabilities
15Scenarios

Capabilities included

The broad abilities this benchmark evaluates. Each capability is defined globally; this page states that it belongs to this benchmark.

CapabilityWhat it means hereScenarios
Quiz GenerationCreates valid quizzes from supplied material that test the intended knowledge and include sufficient information for answers to be evaluated.5
Quiz DeliveryLets learners take and submit quizzes in the intended individual or class setting while recording each attempt correctly.2
GradingDetermines how correctly a learner answered, including fixed-answer scoring, semantically equivalent wording, and partial understanding.3
Answer FeedbackExplains what the learner understood correctly and what remains wrong, missing, or incomplete.2
Personalized PracticeChanges later practice according to the learner's demonstrated strengths and weaknesses, including across sessions.2
Learner ProgressShows learners how their performance changes over time and which topics still need work.1
Class ReportsShows teachers how learners performed and which questions or topics the class struggled with.1

Scenarios included

A scenario is a real-world situation used to test a capability. Together these scenarios define the evaluation scope of version 1.

Quiz Generation

5 scenarios
  1. A cleanly scanned document
  2. The user turns study material into a quiz
  3. Important information appears in both text and a diagram
  4. The user asks the quiz to test a particular learning goal
  5. The user turns a recorded lecture into a quiz

Quiz Delivery

2 scenarios
  1. A learner takes a quiz on their own
  2. A teacher gives the quiz to a class

Grading

3 scenarios
  1. The learner answers some fixed-answer questions correctly and others incorrectly
  2. The learner answers correctly in their own words
  3. The learner's answer is partly correctAlso exercises Answer Feedback

Answer Feedback

2 scenarios
  1. The learner's answer is partly correctAlso exercises Grading
  2. The learner gives a clearly wrong answer

Personalized Practice

2 scenarios
  1. One topic is weaker than the others
  2. The learner returns to the material later

Learner Progress

1 scenario
  1. The learner checks whether they are improving

Class Reports

1 scenario
  1. The teacher checks what the class struggled with

How the benchmark works

A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.

Scope

Scope includes quizzes and practice tests

The benchmark covers quizzes and practice tests from supplied study material, then delivering, grading, explaining, and tracking them.

Method boundary

Open rubrics and scoring

Rubrics, weights, scoring, and ranking methodology remain open and unpinned.

Separate attributes

Product attributes are separate

Product attributes are captured separately and never scored.

Class reports

Class reports show struggle, not trend

Class reports cover what the class struggled with; do not invent an over-time class-performance trend.

Test case implementation and fixture design remain the open stage.

Resources and fixtures

The registered material and systems that create a consistent test environment for this benchmark.

Quiz Generation — Benchmark definition | AI Demos