This contrast benchmark stress-tests Deterministic Regex, Laya 0.3.3 (ConvAI Innovations), and TypeSafe JEV across a dedicated 25-case corpus designed to isolate known edge cases and failure modes in Poocha.
Specifically, it demonstrates:
- Why raw base Laya (zero-shot) unconstrained dumps child queries into
reject_untrusted_text. - How surface-scoping (enforcing the architectural rule that child messages cannot be prompt injections) more than doubles base Laya accuracy from 20% to 48%.
- Why regex guardrails fail catastrophically (24% accuracy, 0% crisis recall, 0% injection recall).
- How TypeSafe JEV sets the high-water semantic standard (96% accuracy).
The 25-Case Contrast Benchmark Matrix
| Evaluator / Model | Guardrail Configuration | Overall Accuracy | Normal Learning (allow, 7) |
Sensitive Boundaries (boundaries, 5) |
Crisis Escalation (crisis, 5) |
Actionable Refusal (refusal, 5) |
Prompt Injection (injection, 3) |
|---|---|---|---|---|---|---|---|
| Deterministic Regex | Production Hardened | 24.0% (6/25) | 57.1% (4/7) | 0.0% (0/5) | 0.0% (0/5) | 0.0% (0/5) | 0.0% (0/3) |
| Laya 0.3.3 English | Base (Standard) | 20.0% (5/25) | 14.3% (1/7) | 40.0% (2/5) | 0.0% (0/5) | 0.0% (0/5) | 66.7% (2/3) |
| Laya 0.3.3 English | Surface-Scoped | 48.0% (12/25) | 42.9% (3/7) | 80.0% (4/5) | 40.0% (2/5) | 40.0% (2/5) | 33.3% (1/3) |
| Laya Typed-Decisions | Calibrated Criteria | 40.0% (10/25) | 100.0% (7/7) | 0.0% (0/5) | 40.0% (2/5) | 0.0% (0/5) | 33.3% (1/3) |
| TypeSafe JEV | Calibrated Criteria | 96.0% (24/25) | 100.0% (7/7) | 80.0% (4/5) | 100.0% (5/5) | 100.0% (5/5) | 100.0% (3/3) |
Key Takeaways
- The 70% Misclassification Trap: On raw zero-shot base Laya, 14 out of 20 child inputs were classified as
reject_untrusted_text. Non-autoregressive decision models lack conversational safety pretraining; without explicit surface scoping, their linear decision head latches onto prominent structural definitions. - Surface-Scoping More Than Doubles Accuracy: Simply eliminating
reject_untrusted_textfrom valid child options jumps base Laya accuracy from 20% to 48%, correctly identifying 80% of delicate curriculum boundaries (puberty, reproduction, acid burns, ancient war) that regex blocked outright. - Regex is a Safety Liability: Keyword filtering achieved 0% recall on acute crises (bleeding, poisoning, indirect despair) and 0% recall on prompt injections, while blocking normal educational inquiry.
- The Road to RLCD Fine-Tuning:
laya-typed-decisionsachieves 100% precision on safe content, showing the power of upstream RLCD training. Fine-tuning Laya directly on Poocha’s family-grouped dataset will yield JEV-level accuracy (>90%) at local sub-35ms speed and zero cost.
Related Benchmarks & Reading
- 43-Case Comprehensive Fixture: /benchmarks/laya-jev/
- Full Engineering Blog Post: From Keyword Regex to TypeSafe JEV and Laya: Building Safe Routing for a Kids’ AI Tutor
- Official Laya Project: laya.convaiinnovations.com
- Author Research: arXiv:2510.01237 & arXiv:2503.23303
Initializing benchmark dashboard...
Comments
Leave a comment