Scenario in benchmark · Managed RAG · Version 1

A bad file in the upload

A bad file uploaded for ingestion cannot be processed.

Capability: Data Ingestion1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the service reports failure for the uploaded file.
  2. Whether the file is marked as not processed or not ingested.
  3. Whether the upload is presented as unsuccessful rather than successful.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
Denser AIFailed1 of 1 test case failedReported an unreadable truncated PDF as ready rather than flagging its failure.View Result →

It did flag a password-protected PDF as failed in the same upload.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowFailed1 of 1 test case failedDid not flag the two unreadable uploads as document failures.View Result →

A truncated PDF and a password-protected PDF received the same success status as a readable comparison document.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
DifyPassed1 of 1 test case passedReports both unprocessable uploads as errors.View Result →

Each file ends in error with its own PDFium message, and neither is completed or silent.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this upload scenario: it reported both bad PDFs as failures instead of ingesting them.View Result →

The truncated file and the password-protected file were both rejected; the app said each could not be uploaded, and the API returned HTTP 500.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedReported files it could not process during upload.View Result →

It returned errors for a truncated PDF and a password-protected PDF; the console also showed the truncated file's failure.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Data Ingestion

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A bad file in the upload in Managed RAG | AI Demos