Skip to content

Evaluation & Results

Review the Primary Care evidence and download both result tables.

Separate checks, each with its original date and denominator. These results are not combined into one pass rate, and all data is synthetic.

How we evaluate

1 · DesignGold questions, synthetic fixtures, tool contracts, refusal cases
→
2 · ExecuteFastMCP fixtures, IBM watsonx Orchestrate, DeepEval, voice trials
→
3 · CompareExpected vs. observed, with failures and evidence limits kept

Results at a glance

Area Date Scope Result Details
Gold questions Nov 19 Local Primary Care fixture 16/16 matched Method & gold
Control checks Nov 19 Six deterministic local suites 135/135 passed Method & gold
MCP contracts and regression Nov 15–16 Tool schemas, scope, backend tests passed 1 Windows-only failure (Nov 18) Validation
IBM and DeepEval Oct–Nov Bounded agent and voice runs recorded metadata gaps Agent evidence
End-to-end booking Nov 16 One synthetic voice trial at a pinned revision BOOKED_SIMULATED End-to-end
Security and adversarial Nov 15–19 Probes, voice sample, reconciliation bounded; gaps open Security

Timeline

OCT 2026Baseline10-question gold set, M6.3 runs
NOV 15–16Server validationContracts, probes, regression
NOV 16Voice trialSynthetic booking receipt
NOV 19Judge review16 gold answers, 135 checks
Reproduce a read-only check

Use the Developer Guide for the local server, MCP Inspector, IBM CLI, DeepEval and optional Phoenix setup.

{"name":"get_scheduling_encounter","arguments":{"case_id":"DEMO-PC-5000999-001"}}

This fixture returns structured scheduling evidence and does not book an appointment. The BOOKED_SIMULATED receipt belongs to the separate, dated voice trial.

Scope

A registered tool is not proof that every operation was exercised. No unverified Phoenix score is shown. Links to evidence repositories may require reviewer access.

Next: Method & gold questions · Demonstration