Scenario in benchmark · Managed RAG · Version 1
A bad file in the upload
A bad file uploaded for ingestion cannot be processed.
What we check
- Whether the service reports failure for the uploaded file.
- Whether the file is marked as not processed or not ingested.
- Whether the upload is presented as unsuccessful rather than successful.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIFailed1 of 1 test case failedReported an unreadable truncated PDF as ready rather than flagging its failure.View Result →
It did flag a password-protected PDF as failed in the same upload.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowFailed1 of 1 test case failedDid not flag the two unreadable uploads as document failures.View Result →
A truncated PDF and a password-protected PDF received the same success status as a readable comparison document.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedReports both unprocessable uploads as errors.View Result →
Each file ends in error with its own PDFium message, and neither is completed or silent.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this upload scenario: it reported both bad PDFs as failures instead of ingesting them.View Result →
The truncated file and the password-protected file were both rejected; the app said each could not be uploaded, and the API returned HTTP 500.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedReported files it could not process during upload.View Result →
It returned errors for a truncated PDF and a password-protected PDF; the console also showed the truncated file's failure.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Data Ingestion
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →