Sample deliverable · AI & Automation
Sample architecture review: RAG assistant scorecard
How we score a GenAI system across retrieval, evaluation, security, operations and cost, with example findings.
Sample / template. Fictional systems and figures that show the format and depth of what you receive. Not a client deliverable.
System: internal policy assistant (fictional) · Stack: managed LLM, pgvector, document store · Rating scale: 1 (missing) to 4 (production-ready)
Scorecard
| Area | Score | Summary |
|---|---|---|
| Use case and success measures | 2 | Users clear; no agreed quality threshold |
| Evaluation | 1 | Manual spot checks only |
| Retrieval quality | 2 | Fixed-size chunks; no hybrid search or re-ranking |
| Permissions | 1 | All users can retrieve all documents |
| Safety and guardrails | 2 | Basic system prompt; no injection tests |
| Observability | 2 | Request logs, no retrieval traces |
| Cost control | 3 | Per-request limits; no daily cap |
Top findings
- Permissions not enforced at retrieval. Documents restricted in the source system are retrievable by any user. Must fix before launch.
- No evaluation set. Quality cannot be measured, so changes cannot be approved with confidence.
- Retrieval misses tables and headings. Structure-aware chunking and hybrid search are likely to improve groundedness.
Recommendations and sequence
| Step | Work | Effort |
|---|---|---|
| 1 | Build a 200-question evaluation set with experts | 1–2 weeks |
| 2 | Enforce document permissions at retrieval | 1 week |
| 3 | Chunking, hybrid search and re-ranking experiments, scored | 2 weeks |
| 4 | Guardrails, injection tests, tracing and daily spend caps | 1–2 weeks |
Release criteria (proposed)
- Evaluation score at or above the agreed threshold on two consecutive runs
- Zero permission leaks in the access test suite
- Tracing and spend caps live in production
Want this for your systems?
Every paid assessment ends in a document like this, with your real systems, risks and a fixed quote. The fee is credited to the project.