Validation & Agent Evidence
Backend validation, agent runs and the end-to-end trial. Backend authorization correctness is assessed separately from agent behavior.
MCP server validation
Nov 15–16 checkpoint. Deterministic assertions with no model: tool contracts, scoped inputs, explicit denials and backend reliability, run in local fixture mode.
| Run | Result | Boundary |
|---|---|---|
| Nov 15 backend regression | 1,337 passed; 19 skipped | Earlier release |
| Nov 16 Linux regression | 1,380 passed; 19 skipped | Separate Linux run |
| Nov 16 contract probes | 108/108 | Recorded 25-tool configuration |
| Nov 16 hostile-argument probes | 76/76 | Input checks, not a prompt-injection certification |
| Nov 16 scope and schema | 261/261 | Overlapping assertions; do not sum with probes |
| Nov 18 local Windows review | 1,624 passed; 19 skipped; 1 failed | Linux-only os.O_NOFOLLOW is unsupported on Windows |
The sample fixture is read-only, and an unknown case returns a denial instead of invented scheduling facts:
{"name":"get_scheduling_encounter","arguments":{"case_id":"DEMO-PC-5000999-001"}}
Evidence: Nov 16 tool-wide report · Per-tool coverage · Contract probe source · Argument probe source
IBM watsonx Orchestrate and DeepEval
Assesses agent behavior, MCP tool selection and spoken interaction using historical M6.3 and M6.4 configurations. Model metadata is disclosed only from the exact run artifact.
| Source | Recorded result | Interpretation |
|---|---|---|
| Historical M6.3 saved-agent run | 7/7 | Historical configuration only |
| Nov 15 DeepEval rerun | 0.95 MCP-use reading | One conversation: two tasks, three calls |
| Nov 16 controlled voice call | Five IBM-selected tool types observed | Synthetic booking path; backend authority enforced separately |
| Nov 18 spoken identity check | PASS identity only | No appointment requested; not a booking rerun |
Evaluator metadata
The reviewer summary does not establish a single reusable DeepEval judge-model ID, temperature, rubric, prompt or threshold for every run. Do not insert a guessed model name. An invalid earlier zero-score adapter run is not evidence that Avery failed every task.
IBM CLI commands (historical runners)
Use an authorized IBM environment and inspect the selected toolkit first.
orchestrate --version
orchestrate env list
$EnvironmentName = 'YOUR_AUTHORIZED_ENVIRONMENT'
orchestrate env activate $EnvironmentName
orchestrate agents list
orchestrate toolkits list
pwsh ./deploy/ibm/orchestrate/run-m63-evaluation.ps1 -EnvironmentName $EnvironmentName
pwsh ./deploy/ibm/orchestrate/run-m64-evaluation.ps1 -EnvironmentName $EnvironmentName
These scripts target historical agent and toolkit names and do not evaluate the newer consolidated Avery configuration.
Evidence: IBM CLI setup · Historical release evaluation · Evaluation assets
End-to-end voice trial
BOOKED_SIMULATED Nov 16, 2026, at a pinned revision. Generated caller audio exercised IBM-selected patient retrieval, knowledge search, appointment search, option selection and booking request tools. Trusted services independently checked scope and current confirmation. Database reads confirmed one appointment and one host-written CRM note, with no production VA writes. This is not a new Nov 19 booking trial.
Added by later reviews
- Nov 18: spoken identity and caption-masking acceptance (identity only; no booking).
- Nov 19: 16/16 gold answers and 135/135 deterministic control checks.
- Nov 19: voice and database reconciliation found gaps in before/after snapshots for earlier attack sessions.
What remains open
Open evidence areas
| Area | Required evidence |
|---|---|
| Later revision acceptance | A separately authorized run on fresh synthetic cases if the release changes |
| Security | Expanded attack coverage, paired DB checks, distinct-session isolation, dependency triage |
| Reliability | Hosted multi-writer concurrency and uncertain-write recovery checks |
| Delivery | Final recorded browser presentation; visual check of receipt and letter flow |
| Reproduction | Pinned final release and locked dependency freshness |
Evidence: Accepted booking proof · Accepted call report · Nov 19 reviewer record
Next: Security & adversarial