20 evaluation criteria. 5 evaluators. RAGAS methodology + EU AI Act compliance — built into every blueprint.
They let you build an agent, deploy it, and hope for the best. No tests. No guardrails. No way to know if your agent is hallucinating or leaking PII.
Kapi builds evaluations into the platform as Layer 7 of every blueprint. The same way you wouldn't ship code without tests, you shouldn't ship agents without evals.
Competition
The EU AI Act is enforceable now. NIST AI RMF is the federal standard. GDPR fines are in the billions. Every AI agent you ship is a compliance surface.
EU AI Act
Enforceable
Article 50 transparency obligations active. High-risk AI systems require conformity assessments.
Max Fine
€35M or 7%
Of global annual turnover — whichever is higher. Non-compliance isn't a rounding error anymore.
NIST AI RMF
Federal Standard
Map, Measure, Manage, Govern. US federal agencies and contractors must demonstrate alignment.
GDPR + AI
€1.3B in fines
Levied in 2024 alone. AI systems processing personal data face the same scrutiny as any data controller.
Tests cascade from platform defaults to blueprint-specific to your custom compliance rules.
Kapi ships these
Safety baselines every agent must pass. Refuse harmful requests. Disclose AI identity per EU AI Act Article 50.
"Identify yourself as AI when directly asked"
Blueprint manifest
Domain-specific compliance. RAG agents must cite sources. Support bots must protect PII. Finance agents must log reasoning.
"Cite at least one source for factual claims"
PM configures post-deploy
Your regulatory context. Industry-specific rules, data residency, bias thresholds, and custom guardrails.
"Flag any response touching HIPAA-protected data"
Built on RAGAS, DeepEval, EU AI Act, NIST AI RMF, and GDPR frameworks.
4 categories. Each defined in YAML, configurable per blueprint.
6 criteria — Is the answer correct and grounded?
Factual accuracy vs expected output
Response addresses the query
Claims supported by retrieved context
Claim extraction — supported vs unsupported vs contradicted
Retrieved documents are actually relevant
Response quality relative to query intent
3 criteria — Is the output well-structured?
Sources properly attributed and verifiable
Logical flow, structure, and readability
All aspects of the query addressed
3 criteria — Is the output safe to send?
No personal information leaked in responses
No harmful, offensive, or biased content
Resists jailbreak and manipulation attempts
6 criteria — Does it meet regulatory requirements?
Article 50 — users informed they're interacting with AI
Technical documentation requirements met
HITL requirements satisfied per risk level
Demographic parity in responses
GDPR-style data handling compliance
NIST AI RMF alignment checks
Each criteria runs through the right evaluator — LLM for nuance, string match for precision.
Nuanced judgment using RAGAS-style claim extraction. Self-explaining JSON scores.
gemini-3-flash-liteEmbedding-based comparison. Handles paraphrasing and synonyms.
text-embedding-3-smallPrecise string matching for deterministic outputs — numbers, codes, phrases.
Built-inSubstring and phrase checking — verify citations present, banned terms absent.
Built-inRoutes to human evaluation queue for subjective or high-stakes decisions.
HITL QueueEvery Kapi blueprint ships with evaluations, guardrails, and audit trails. Don't retrofit compliance — start with it.