Here's a question that should make every AI builder uncomfortable: would you ship code without tests?

Of course not. You'd have unit tests, integration tests, CI/CD pipelines, staging environments. The idea of pushing untested code to production is absurd.

And yet, that's exactly what most AI agent platforms encourage. Build an agent, hit deploy, cross your fingers. No evaluation framework. No quality metrics. No safety checks. No compliance verification.

We analyzed the competitive landscape. Out of 9 major platforms — Dify, CrewAI, n8n, Gumloop, and others — none ship a comprehensive evaluation framework. Most have zero eval capabilities. A few offer basic metrics. None come close to what enterprise deployment actually requires.

At Kapi, evaluations aren't an add-on. They're Layer 7 of every blueprint — baked in from day one.

The Four Categories

We organized our 18 evaluation criteria into four categories, each addressing a different dimension of agent quality:

Core (6 criteria) — "Is the answer correct?"

These are the foundational quality checks every AI agent needs:

Correctness — Does the response actually answer the question? An LLM judge evaluates whether the agent's output matches the expected behavior.

Relevance — Is the response on-topic? Agents that go off on tangents erode user trust fast.

Groundedness — Are the claims based on provided context rather than hallucinated? This is critical for any knowledge-based agent.

Faithfulness (RAGAS) — This is our implementation of the RAGAS methodology. The evaluator extracts every factual claim from the response, then checks each one against the source context:

SUPPORTED: Claim appears in the context
UNSUPPORTED: Claim doesn't appear in the context
CONTRADICTED: Claim directly opposes the context

The faithfulness score = Supported Claims / Total Claims. A threshold of 0.85 means at least 85% of claims must be grounded in source documents.

Context Precision — Are the retrieved documents actually relevant to the query? Poor retrieval means poor answers, regardless of how good the LLM is.

Answer Relevance — Does the response address the specific question asked? Different from correctness — an answer can be factually correct but not relevant to what was asked.

Quality (3 criteria) — "Is the output well-structured?"

Citations — Does the response include source references? For enterprise knowledge agents, citations are mandatory. Users need to verify claims.

Coherence — Is the response logically structured and easy to follow? LLM outputs can be technically correct but poorly organized.

Completeness — Does the response cover all aspects of the query? Partial answers are often worse than admitting "I don't have enough information."

Safety (3 criteria) — "Is the output safe to send?"

PII Detection — Does the response contain personally identifiable information that shouldn't be exposed? Names, emails, phone numbers, SSNs — the evaluator catches these before they reach the user.

Toxicity — Is the response free from harmful, offensive, or inappropriate content? Even when the input is adversarial.

Prompt Injection Defense — Has the user attempted to manipulate the agent into ignoring its instructions? The evaluator detects common injection patterns and blocks the response.

Compliance (6 criteria) — "Does it meet regulatory requirements?"

EU AI Act Transparency — Does the agent clearly identify itself as AI? Does it explain its capabilities and limitations?

EU AI Act Documentation — Is the agent's decision-making process documented and explainable?

Human Oversight — Are high-risk decisions routed through human review before execution?

Bias & Fairness — Does the agent produce equitable outputs across different demographic groups?

Data Protection — Does the agent handle personal data in compliance with GDPR and other privacy regulations?

NIST AI RMF Trustworthiness — Does the agent meet the National Institute of Standards and Technology framework for AI risk management?

The 5 Evaluator Engines

Not every criterion can be evaluated the same way. Kapi ships 5 distinct evaluator engines:

Engine	Best For	Speed	Example
LLM-as-Judge	Nuanced judgment, reasoning quality	~2s	Faithfulness, coherence, bias
Semantic Similarity	Comparing meanings, paraphrase detection	~200ms	Context precision, answer relevance
Exact Match	Deterministic outputs, codes, numbers	<10ms	Specific field extraction
Contains	Presence/absence of phrases or patterns	<10ms	Citation checking, PII detection
Human Review	Subjective criteria, high-stakes decisions	Variable	Compliance verification

The LLM-as-Judge engine uses Gemini 3 Flash Lite for cost-effective evaluation at scale. It produces self-explaining JSON scores — not just a number, but detailed reasoning for why the score was assigned.

3-Tier Golden Test Hierarchy

Evaluations need test cases. Kapi provides a three-tier cascade:

Tier 1 — Base tests (Kapi ships these) Generic capability tests every agent should pass. Can it handle a greeting? Does it refuse harmful requests? Is it coherent? These catch fundamental failures before your custom logic even runs.

Tier 2 — Blueprint tests (from the blueprint manifest) Tests specific to your blueprint type. A RAG agent must cite sources. A support bot must check order status correctly. A contract reviewer must extract all required fields. These are defined in the blueprint spec and validate that the agent fulfills its core function.

Tier 3 — Tenant tests (PM adds post-deploy) Your custom business rules. Never mention a competitor by name. Always recommend scheduling a call. Match your brand's tone of voice. These are the rules that make your agent yours.

The cascade means every interaction is checked against all three tiers. A failure at any level can trigger alerts, HITL escalation, or response blocking.

What This Looks Like in Practice

When you deploy a Kapi blueprint, evaluations run automatically on every interaction. Here's what the PM sees:

Real-time dashboard with pass/fail rates across all 18 criteria
Trending charts showing quality scores over time (drift detection)
Failure details with the exact input, output, and evaluator reasoning
Threshold alerts when any criterion drops below its configured minimum

No configuration required for the basics. Set it and forget it — unless something goes wrong, in which case you'll know immediately.

The Competitive Reality

Here's where the market stands on evaluations:

Platform	Eval Criteria	Evaluators	Compliance	Golden Tests
Gumloop	0	0	None	No
Dify	0	0	None	No
n8n	0	0	None	No
CrewAI	Basic	1	None	No
Kapi	18	5	EU AI Act + NIST	3-tier

This isn't a minor feature gap. It's a category-defining difference. Platforms without evaluations are asking you to deploy AI agents blind.

Get Started

Every Kapi blueprint includes all 18 evaluation criteria, all 5 evaluator engines, and the 3-tier golden test hierarchy. Configure thresholds, add custom business rules, and monitor quality from day one.

Explore the full eval system with interactive examples or start building.

Built on RAGAS methodology, DeepEval self-explaining patterns, and EU AI Act compliance criteria. Learn more about our research foundation.

18 Eval Criteria Your AI Agent Needs Before Going to Production