Next platform deadline: Amazon EKS Nov 26, 2026  

All 14 deadlines

Sample deliverable · AI & Automation

Sample architecture review: RAG assistant scorecard

How we score a GenAI system across retrieval, evaluation, security, operations and cost, with example findings.

Sample / template. Fictional systems and figures that show the format and depth of what you receive. Not a client deliverable.

System: internal policy assistant (fictional) · Stack: managed LLM, pgvector, document store · Rating scale: 1 (missing) to 4 (production-ready)

Scorecard

AreaScoreSummary
Use case and success measures2Users clear; no agreed quality threshold
Evaluation1Manual spot checks only
Retrieval quality2Fixed-size chunks; no hybrid search or re-ranking
Permissions1All users can retrieve all documents
Safety and guardrails2Basic system prompt; no injection tests
Observability2Request logs, no retrieval traces
Cost control3Per-request limits; no daily cap

Top findings

  1. Permissions not enforced at retrieval. Documents restricted in the source system are retrievable by any user. Must fix before launch.
  2. No evaluation set. Quality cannot be measured, so changes cannot be approved with confidence.
  3. Retrieval misses tables and headings. Structure-aware chunking and hybrid search are likely to improve groundedness.

Recommendations and sequence

StepWorkEffort
1Build a 200-question evaluation set with experts1–2 weeks
2Enforce document permissions at retrieval1 week
3Chunking, hybrid search and re-ranking experiments, scored2 weeks
4Guardrails, injection tests, tracing and daily spend caps1–2 weeks

Release criteria (proposed)

  • Evaluation score at or above the agreed threshold on two consecutive runs
  • Zero permission leaks in the access test suite
  • Tracing and spend caps live in production

Want this for your systems?

Every paid assessment ends in a document like this, with your real systems, risks and a fixed quote. The fee is credited to the project.