When generation gets cheap, evaluation becomes the edge.
The Taste Layer = Quality Gates for AI Outputs
Three technical capabilities: Correctness (accuracy, factuality) + Safety & Compliance (PII, regulations) + Brand Quality (tone, style, trust)
Modern AI excels at fluency—but fluency isn't accuracy.
Simone Leonelli — 16 years building quality systems for luxury brands where defects were unacceptable (Fendi, Bentley, Porsche). After seeing enterprises struggle with AI quality, I spent 2024-2025 bridging design systems thinking and ML evaluation.
Now building AI evaluation infrastructure for high-stakes enterprise applications.
CASE STUDY: Moffatt v. Air Canada (2024)
A simple customer support chatbot without guardrails promised a refund policy that didn't exist. When sued, the airline argued the AI was a "separate legal entity" responsible for its own actions.
The tribunal rejected this. The company was held liable for the hallucination—specifically for negligent misrepresentation via incorrect policy guidance.
Lesson: This case involved a basic chatbot without modern evaluation layers. It demonstrates what happens when AI outputs reach customers without systematic quality checks.
Source: McCarthy Tétrault Legal Analysis
I've analyzed 10+ enterprise AI deployments across healthcare, finance, and B2B services.
The majority lacked dedicated evaluation gates.
Observation from consulting engagements: "Evaluation layer" defined as systematic gating of outputs before customer delivery, with defined metrics, ownership, and escalation paths.
The Industry Shift: The bottleneck in AI development has shifted from "can we generate this?" to "can we trust this?" Evaluation infrastructure is the unsexy but critical layer that separates production-ready AI from expensive demos.
What "Taste Layer" Actually Means:
Taste Layer = Measurable Quality Constraints + Automated Gates + Audit Trails.
It is the infrastructure that makes AI outputs shippable—not subjective aesthetics, but rigorous adherence to policy, correctness, and brand compliance.
The Common Failure Mode: Ad-Hoc Evaluation
Many organizations rely on "vibes-based" spot checks rather than systematic gating. The AI team tracks velocity, but ownership of accuracy falls through the cracks between Engineering (who build pipelines) and Product/Legal (who define quality).
Without a dedicated evaluation layer, the customer becomes the QA team.
The AI industry delivered "generation at scale."
But without quality checks, volume creates liability, not value.
We are living through a constraint shift: generation got cheap and fast, evaluation stayed expensive and slow.
"Marginal cost of generation is collapsing" is directionally right, but technically sloppy.
The truth is scarier:
*Directional trends based on LLM API pricing (Epoch AI, 2020-2025) and enterprise evaluation infrastructure costs. Projections beyond 2025 are directional estimates, not forecasts. Actual costs vary by implementation.
Traditionally, judgment was artisanal — slow, expensive, unscalable. One expert could train maybe 10 apprentices in a lifetime.
In AI, we can BUILD judgment systems. Scale them. Prove them. Deploy them at machine scale.
This is the arbitrage.
The Methodology Transfer:
In high-stakes manufacturing, products pass through multiple independent quality gates—materials, construction, and final review. No single person can approve their own work.
This principle applies to AI: the system that generates should never be the system that judges. Separation of concerns isn't just good architecture—it's how you build trust at scale.
Technically: Judgment is a trained compression function over a domain. In enterprise terms: Domain-Aligned Quality.
"Judgment can be instant, but it is rarely free. Someone paid for it in years."
In markets, rents concentrate at constraints.
Adapted from quality engineering:
Engineering, Legal, and Product have different budgets and incentives.
| Stakeholder | What They Care About | Budget Source | Your Pitch |
|---|---|---|---|
| Engineering/CTO | Ship velocity, incident rate | Platform/Infra | "Reduce on-call incidents from AI failures" |
| Risk/Legal/CISO | Liability exposure, compliance | Risk budget | "Auditable defense against AI liability" |
| Product/Brand | Customer trust, NPS | Product budget | "Quality as differentiator, not cost center" |
Most companies treat evaluation like a post-ship audit. Winners build rejection gates before anything reaches production.
trace_9f8a7d2 (Stored for Audit)
This is not just a philosophy. It is a set of tangible artifacts that your engineering, legal, and product teams will own.
{
"CRITICAL": {
"HALLUCINATION_FACT": "Claims contradiction with retrieval ctx",
"UNSAFE_PII": "Contains email/phone/SSN",
"ILLEGAL_ADVICE": "Offers medical/legal judgment"
},
"WARN": {
"OFF_BRAND_TONE": "Tone mismatch score > 0.4",
"VERBOSITY": "Response > 150 words"
}
}
class FactJudge(dspy.Signature):
"""Judge if response is supported by context."""
context = dspy.InputField()
response = dspy.InputField()
# Output
is_supported = dspy.OutputField(desc="True/False")
citation_check = dspy.OutputField(desc="List of missing citations")
We treat the judge as a software product that requires its own testing. This is how we ensure reliability.
We measure agreement between AI Judge and human experts. If they disagree, we refine the rubric.
Run judges against a static "Control Set" to ensure standards don't shift with model updates.
Adversarial testing to ensure the judge catches subtle errors and isn't fooled by fluency.
Every rubric and judge prompt is versioned. Decisions are traceable to specific criteria.
Companies deploying AI without systematic quality checks are accumulating hidden risk. The Taste Layer is how leading enterprises are building durable competitive advantage.
The Taste Layer is not universally required. Here's how to know if it's right for your use case.
The Taste Layer = 3 Technical Capabilities "Taste" isn't subjective—it's three measurable evaluation systems working together:
Factuality, task success, technical accuracy. Measurable via hallucination rate, accuracy metrics.
PII detection, regulatory alignment, policy adherence. Measurable via violation rates, audit results.
Tone, style, UX fit, customer experience. Measurable via brand alignment scores, trust metrics.
Each layer has distinct metrics, tooling, and funding sources (Engineering, Risk/Legal, Brand/Product). "Taste" is the brand wrapper for these three technical capabilities.
Concrete examples of evaluation gates by vertical:
Healthcare
Gate 1: Block medical advice outside scope of practice
Gate 2: Verify clinical claims against FDA-approved indications
Gate 3: Ensure HIPAA-compliant language
Finance
Gate 1: Detect and block PII leakage
Gate 2: Validate regulatory disclosures (FINRA, SEC)
Gate 3: Check investment advice against fiduciary standards
Retail/Brand
Gate 1: Filter offensive or off-brand language
Gate 2: Verify product claims against inventory/specs
Gate 3: Enforce brand voice guidelines and tone
What an enterprise buyer expects to see on Monday.
Core Infrastructure Components
⚙️ EVAL OPS IS REAL WORK:
Evaluation is a production system with ownership, on-call rotation, and versioning. It's not a script. Treat the judge like any critical service: monitor uptime, track drift, test with adversarial prompts, maintain human panels for calibration.
System Architecture: Component View
Input
User prompt
+ Context
+ History
Generator
LLM API
+ Retrieval
+ Tools
Eval Gates
Judge(s)
+ Policy filters
+ Golden set
Routing
Pass
Auto-fix loop
Human review
Supporting Infrastructure
Telemetry: Log all prompts, outputs, scores, model versions
Dashboard: Defect rate, coverage, latency, cost tracking
Human Queue: Escalated cases, calibration, golden set curation
→ Flow: Input passes through generator, eval gates score output, routing logic decides pass/fix/escalate, telemetry captures everything
Correctness: Factual error rate, task success, regression pass rate, citation validity
Safety/Compliance: PII leak rate, policy violation rate, refusal accuracy
Brand/Product: Tone match score, style violations, editor accept rate, complaint rate
Week 1: Collect 200-500 real outputs, label defects, build golden set v0, choose metrics, set SLOs
Week 2: Implement gates 1-2 (safety + correctness), dashboard defect rate, create escalation queue, run canary release
⚠️ Addressing the Objection "LLM-as-a-Judge is unreliable. Judges are biased, brittle, and easy to prompt-hack."
This is a valid concern. The solution is not to trust the judge blindly, but to treat it like any production system requiring testing, monitoring, and calibration.
Mitigation Strategies
Ownership & Monitoring
The judge is tested and monitored like any model. It's not magic—it's infrastructure.
While providers (Anthropic, OpenAI) solve safety and tooling platforms (LangSmith, Arize) solve tracing, domain-specific judgment remains the enterprise gap.
* Visual hypothesis illustrating the strategic shift from technical assembly to quality assurance.
Evaluation is a production system with real trade-offs.
Robust judges take time.
Verdict: Essential for async/email. Skip for real-time voice.
Strict gates can kill creative outliers.
Verdict: Tune "Judge Temperature" to match risk tolerance.
Golden sets rot as models change.
Verdict: Requires an "Eval Ops" owner, not just a script.
Generic checks (PII, Syntax) will become free/standard.
Verdict: The value is in bespoke Policy & Brand definitions.
| Indicator | Observed Trend |
|---|---|
| The Talent Filter | Ratio of "AI Engineer" roles requiring "Evaluation/Red-teaming" skills will flip from minority to majority by 2027. |
| The Compute Flip | Compute investment is shifting toward Inference-Time Reasoning as the primary lever for accuracy and reliability. |
| The New Category | Evaluation Infrastructure is emerging as a standalone category, independent of foundation model providers. |
Falsifiable Prediction By December 2026, at least one Fortune 500 company will list "AI Evaluation," "Model Risk," "AI Assurance," or "GenAI Quality" as a core competency in their 10-K filing, annual report, or earnings call transcript.
If I'm wrong, the manifesto ages poorly. If I'm right, you heard it here first. Hold me accountable.
What counts: Public disclosure in SEC filings (10-K), annual reports, risk sections, or earnings transcripts explicitly referencing evaluation capability as strategic or operational competency.
🔬 What Would Falsify This Thesis
Intellectual honesty requires stating what could prove me wrong. Here are concrete scenarios that would undermine the need for separate evaluation infrastructure:
Falsifier 1: Platform Commoditization
Foundation model providers ship end-to-end evaluation and governance that removes the need for separate infrastructure for most enterprises.
Falsifier 2: Near-Zero Eval Cost
Tooling advances reduce eval cost to near-zero even for domain-specific judgment, eliminating the economic moat.
Falsifier 3: Human-in-the-Loop Dominance
Enterprises shift to human-in-the-loop workflows at scale instead of automated gates because automated eval remains too brittle.
If any of these scenarios materialize, the strategic value of separate evaluation infrastructure diminishes significantly. I'm tracking these possibilities and will update this thesis accordingly.
The principles of quality assurance are universal, whether applied to physical goods or probabilistic software. Studio W230 translates rigorous quality methodologies—proven in high-stakes manufacturing—into the domain of AI evaluation. This is not about aesthetics; it is about systemizing judgment to prevent defects at scale.
⚡ Do This Tomorrow
The 5-minute test: Take your last 10 AI-generated customer communications. Have someone outside your team rate them for accuracy.
What's your defect rate?
If you don't know the answer, you're flying blind. If the answer makes you uncomfortable, that's the data you needed.
This manifesto represents ongoing research on The Taste Layer.
Working on AI evaluation challenges? I'd love to learn from your experience.
Simone Leonelli — Studio W230
A selection of foundational research and industry reports for those looking to build rigorous evaluation systems.
| Domain | Resource |
|---|---|
| Legal Liability | CRT Tribunal — Moffatt v. Air Canada (2024) |
| Cost Dynamics | Epoch AI — LLM Accuracy-Runtime Trade-offs (2025) |
| Market Research | NIQ — CMO Outlook: Guide to 2026 |
| AI Strategy | IBM — The AI-First CMO C-suite Study |
| Trust Economy | Salsify — Consumer Research 2025: The Value of Trust |
| Industry Forecast | Gartner — AI TRiSM (Trust, Risk & Security Management) |
| Risk Assessment | Stat News — UnitedHealth AI Lawsuit (Alleged 90% Error Rate) |
| Foundational | Simon (1971) — Designing Organizations for an Information-Rich World |