Evaluation & Results
Review the Primary Care evidence and download both result tables.
Separate checks, each with its original date and denominator. These results are not combined into one pass rate, and all data is synthetic.
16 / 16Gold answers matchedNov 19 · Primary Care fixture
135 / 135Local control checks passedNov 19 · six suites
1,380Backend tests passedNov 16 · Linux run, 19 skipped
5Open evidence areasSecurity, reliability, delivery
How we evaluate
1 · DesignGold questions, synthetic fixtures, tool contracts, refusal cases
→
2 · ExecuteFastMCP fixtures, IBM watsonx Orchestrate, DeepEval, voice trials
→
3 · CompareExpected vs. observed, with failures and evidence limits kept
Results at a glance
| Area | Date | Scope | Result | Details |
|---|---|---|---|---|
| Gold questions | Nov 19 | Local Primary Care fixture | 16/16 matched | Method & gold |
| Control checks | Nov 19 | Six deterministic local suites | 135/135 passed | Method & gold |
| MCP contracts and regression | Nov 15–16 | Tool schemas, scope, backend tests | passed 1 Windows-only failure (Nov 18) | Validation |
| IBM and DeepEval | Oct–Nov | Bounded agent and voice runs | recorded metadata gaps | Agent evidence |
| End-to-end booking | Nov 16 | One synthetic voice trial at a pinned revision | BOOKED_SIMULATED | End-to-end |
| Security and adversarial | Nov 15–19 | Probes, voice sample, reconciliation | bounded; gaps open | Security |
Timeline
OCT 2026Baseline10-question gold set, M6.3 runs
NOV 15–16Server validationContracts, probes, regression
NOV 16Voice trialSynthetic booking receipt
NOV 19Judge review16 gold answers, 135 checks
Reproduce a read-only check
Use the Developer Guide for the local server, MCP Inspector, IBM CLI, DeepEval and optional Phoenix setup.
{"name":"get_scheduling_encounter","arguments":{"case_id":"DEMO-PC-5000999-001"}}
This fixture returns structured scheduling evidence and does not book an appointment. The BOOKED_SIMULATED receipt belongs to the separate, dated voice trial.
Scope
A registered tool is not proof that every operation was exercised. No unverified Phoenix score is shown. Links to evidence repositories may require reviewer access.
Next: Method & gold questions · Demonstration