Integration acceptance and probe coverage
A green probe is diagnostic evidence, not full certification. The published extension requirements remain in force even where the current probe does not enforce them. Keep the transcript, but also record the authenticated integration checks below before enabling an agent.
Current probe coverage
This audit describes agent-probe at 17db375, reviewed 2026-10-21. Recheck coverage when upgrading; warnings and skipped checks can still produce exit code 0.
| Area | Current diagnostic | Evidence still needed |
|---|---|---|
| Card | Structural and declaration checks | Advertised interfaces match the deployed server and client dialect |
| Skills | One example invocation | Every relevant skill, edge case and entitlement |
| Citations | Presence diagnostic; absence can warn | Required fields, correct numeric association and reproducible source |
| Refusals | Structural checks | Deliberately triggered rules, both Parts and matching card declaration |
| Data source | Declaration shape | Dataset name agrees with runtime citations |
| Views | Declaration checks and URL fetch; responses below 500 can pass | Authenticated usable page, not just a 404 or login redirect |
| Streams | Ordering heuristics | Actual SDK v1/0.3 parsing, complete artifacts, terminal states and disconnect behavior |
| Identity | Machine credential support | Human grant verification, client binding, trust/scope/audience and isolation |
| Tasks and feedback | No complete acceptance coverage | Ownership, persistence, recovery, retry and evidence association |
Run the available diagnostic
The public npm package @upperhand/agent-probe returned 404 during this
audit. Do not make npx availability an onboarding prerequisite. With
access to the source repository, build the reviewed version:
git clone https://github.com/The-Focus-AI/agent-probe.git
cd agent-probe
git checkout 17db3755fac61f610d716a73105d86ab4c0000dc
npm ci
npm run typecheck
npm test
npm run build
node dist/index.js https://agent.example.com --card-only
Replace the example origin with the environment being checked.
--card-only checks discovery, not invocation. For authenticated checks,
use the probe's documented credential configuration after reviewing its
handling; never paste tokens into transcripts or command arguments. If a
safe human-grant path is unavailable in the probe, use the service's
integration harness and record that gap rather than disabling auth.
Acceptance record for each environment
Record the card URL, service commit, SDK version, wire dialect, test time and credential-free results. Mark each item passed, failed or not supported; an untested capability is not passed.
- Discovery: card and health are reachable; absolute interface URLs resolve to this environment, not loopback or a different deployment. Declared capabilities exist; optional views are agent-authenticated.
- Auth: approved machine and human identities succeed. Missing, expired, wrong-audience, wrong-scope and cross-owner requests fail. Test the chosen client-binding profile with two entitled machines.
- Wire: direct and Workbench-coordinated blocking and streaming calls work in each supported dialect. Task precedes updates; artifacts precede completion; Parts and specialist task references survive.
- State: follow-ups send only the new user turn with the retained context. Task polling survives disconnects/restarts without rerunning tools; another caller cannot retrieve the task or history. Exercise advertised cancellation/resubscription, or report explicit lack of support and the available recovery path.
- Output: verify quantitative results against source evidence and extension schemas. Trigger refusals intentionally and preserve their typed Parts without substituting fabricated answers.
- Feedback, when enabled: verify the agreed discovery and payload, retry semantics, original owner and exact specialist task association.
- Deployment: verify persistent storage, model access, production trust/entitlements, operational owner and rollback procedure in that environment. Local fixtures or an orb portal do not certify production.
The onboarding packet defines the service boundary. The identity and feedback drafts identify decisions that remain open; passing today's probe does not ratify them.