Scenario in benchmark · Managed RAG · Version 1

A simple, clearly formatted table

A simple, clearly formatted table is provided for processing.

Capability: Data Ingestion1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the rows and columns remain aligned after processing.
  2. Whether each value stays under the correct header.
  3. Whether the output still reads as a table rather than a flattened or split block of text.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
Denser AIPassed1 of 1 test case passedHandled this simple table lookup correctly.View Result →

It returned £487 for the Extended out-of-hours cell, and the retrieved table also showed the nearby values 352, 615 and 284 without using them as the answer. The notebook capture and the call log showed the same answer.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedPasses this simple table lookup scenario.View Result →

The chat answer returned £487 (excluding VAT), and the supporting passage showed that value in the Extended row under Out-of-hours callout; the retrieval output showed the same table passage.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedPasses this simple-table scenario in the checked case.View Result →

The table shows Extended / Out-of-hours callout at 487, and the grounded answer gives the same amount. The captured table also shows the nearby values in the same section.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedPreserved the structure of a simple table on retrieval.View Result →

Its first result placed the reference value, 487, at the requested row and column; it returned the table rather than a separate answer naming the value.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedCorrectly answered the row-and-column lookup with 487 for an Extended contract’s out-of-hours callout charge.View Result →

The answer cited the passage containing the charges table.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Vectorize.ioPassed1 of 1 test case passedCorrectly read £487 from the Extended cover row and out-of-hours callout column.View Result →

Its highest-ranked passage contained the whole charges table, and its generated answer gave that amount.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (2): Llama Cloud, Onyx.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Data Ingestion

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A simple, clearly formatted table in Managed RAG | AI Demos