Skip to content

Method & Gold Questions

How the evaluation was designed, from the October baseline to the November 19 judge review. Gold-answer matching is kept separate from deterministic control checks; Phoenix grouped saved results and these were not 135 LLM-judged safety prompts.

16 / 16Gold answers matchedPrimary Care fixture
135 / 135Control checks passedDeterministic, local
6Grouped control suitesPhoenix-organized

Baseline (October to early November)

The foundation was a 10-question gold dataset, synthetic encounter fixtures, MCP schema checks and failure cases. Historical analyzer failures are retained rather than converted to passes.

Checkpoint Recorded observation Interpretation
October foundation Ten-question gold dataset committed; MCP evaluation foundation merged Provenance, not a 10/10 pass claim
M6.3 saved-agent run 7/7 historical score; about 7.97 s average Not a current M6.4 result
M6.3 direct run Ten cases executed; six-case regression accepted with qualifications Analyzer and text-match limits retained
M6.4 dataset Thirteen designed scenarios Dataset size is not a pass result

Gold question design

Questions cover established/new classification, requested care, visit format and duration, eligible and excluded slots, wait-standard comparison, identity, and explicit missing-case denial. Several questions read fields from the same tool response, so 16 matched answers are not 16 independent end-to-end bookings.

Worked example: Gold Question #6

Why is slot VISTA-SLOT-PC-5000999-002 excluded?

Gold Question #6: gold test construction and MCP evaluation

Decision item Expected Observed
Eligible false false
Reason 1 APPOINTMENT_INTENT_MISMATCH Matched
Reason 2 PATIENT_CLASSIFICATION_MISMATCH Matched
Reason 3 TEMPLATE_MISMATCH Matched
Recorded result PASS Expected = observed

Phoenix-grouped control checks

Suite Passed What it covered
Trusted Primary Care 47/47 Host assertion, session and case scope, replay, persisted evidence
Protected Primary Care booking 41/41 Consent and booking grant, forged or expired assertion, recovery, no duplicate writes
Scheduling field contract 14/14 Field ownership; no model-derived governed facts
CRM receipt 12/12 Receipt validation and authorized projections
Synthetic Veteran consent 11/11 Verified identity, case binding, current consent
Booking CRM 10/10 Cross-call denial, idempotency, note provenance
Total 135/135 Six parameterized local suites

What this does not prove

These checks support a bounded implementation claim. They are not a penetration-test certification, comprehensive prompt-injection coverage, or independent verification by a Phoenix LLM judge.

Source artifacts

Next: Validation & agent evidence