API Relay Audit

What it does

API Relay Audit probes a third-party AI API relay, mirror, gateway or LLM proxy — the service between you and the model provider. The repository is a Python audit tool; the DSH half is a thin adapter that adds one command, /relay-audit, and runs the bundled audit.py against a relay you choose.

The distinction that matters most: this plugin does not make your agent safer the way a sandbox or permission layer does. It is a diagnostic that sends probe requests to a relay and writes a Markdown report. It checks prompt-side signals (hidden injection, instruction override, prompt extraction, context truncation), response integrity (tool-call and package-command substitution, error-response leakage, Anthropic SSE stream anomalies), and model-identity signals (mismatched self-identification, stream-level model identity, latency variance, upstream channel fingerprint).

The README is direct about the verdict: it is not a safety certificate, inconclusive is never folded into clean, and a model’s natural-language self-identification is a consistency signal, not proof of upstream substitution.

Running it

The documented install line, pinned to an immutable release tag:

dsh plugin --profile web add "github:toby-bridges/api-relay-audit#v2.4.0"
# dsh-cc-tui and other compatible profile-based clients
dsh plugin --profile cc-tui add "github:toby-bridges/api-relay-audit#v2.4.0"

The README writes the ref as a DSH_PLUGIN_REF=v2.4.0 variable. v2.4.0 is a real tag, its distribution-verification record gives the commit as 571b97142c0d22aae305ac2919e45137bc424dcc, and no newer tag existed when this listing was checked. The README warns against master unless you intend to test unreleased changes, and the bundle is prebuilt JavaScript with no prepare or build hook, so a Git install does not need pnpm’s install-time build allowlist.

The bundle patch contributes an api-relay-audit row to the composition, so one command appears on every DSH command-compatible surface:

/relay-audit
/relay-audit --connectivity
/relay-audit --profile web3 --fast-context
/relay-audit --url <URL> --model <claude-model> --credential-ref <DSH_CREDENTIAL_REF>

With no arguments it resolves a target from your current DSH route and runs the full 14-step audit. The adapter owns five options — --url, --model, --credential-ref, --output, --transparent-log — and passes every other token through to audit.py, whose 22 flags stay available; it rejects abbreviations of its own options, and --key and --key-env outright. One gate: the resolved model must identify as Claude, tested against the model id and, where a provider directory exists, its name and description — without a match the command refuses and writes no report. The relay itself may speak the Anthropic or OpenAI protocol.

The three profiles

audit.py takes --profile general|web3|full, default general. general runs all non-Web3 relay checks, with Web3 injection probes gated out. web3 is general plus three wallet-oriented prompt-injection probes: private-key refusal, signed-transaction refusal, transfer guidance. full runs everything.

The 14 steps in order: infrastructure recon, model list, token injection, prompt extraction, instruction conflict, jailbreak, context length, tool substitution, error leakage, stream integrity, Web3 injection, infrastructure fingerprint, latency variance, upstream channel classifier. Steps 3–6 cover prompt injection and extraction; steps 5, 10, 13 and 14 carry the model-substitution signals.

What a run actually does

It sends real, billable requests to the relay, authenticated with your key. There is no dry-run flag. --connectivity sends exactly two low-token requests (one /v1/messages, one /v1/chat/completions, max_tokens: 8 each) and exits with an OK/WARNING/FAILED verdict instead of a risk rating. The full audit is a different order of magnitude: the context-length scan sends progressively longer prompts (default ladder 50K–800K characters; --fast-context reduces it to 10K–200K), and --aggressive-error-probes adds a 256 KB oversized-context probe whose own help text warns it “may incur metered billing on pay-as-you-go relays”.

It does not modify the relay. The probes are POSTs to message endpoints and informational GETs (/, /v1/models, a deliberately nonexistent path); nothing PUTs, PATCHes or DELETEs. Locally it writes only the report and, on request, an append-only JSONL transparent log recording each request’s timestamp, URL, request/response SHA-256 and status code. The adapter defaults the report to .api-relay-audit/reports/api-relay-audit-<timestamp>-<commandId>.md in the session workspace, and rejects --output or --transparent-log paths that leave the workspace or escape it through a symlink.

It reads the DSH environment you already configured. The adapter injects commands, credentials, subprocess, llm and settings, then falls back to the current provider’s configured profile for baseURL and apiKeyEnv and to the session agent’s model; explicit flags win, and any unresolvable fact is named rather than guessed.

How the credential is handled

The adapter resolves the named DSH Credential, requires it to be non-empty, and passes the value to the Python child only through the environment variable API_RELAY_AUDIT_KEY, invoked as audit.py --key-env API_RELAY_AUDIT_KEY. Command arguments never carry the value: raw --key and caller-supplied --key-env are rejected, /relay-audit is registered with recordInput: false, and any secret found in captured stdout or stderr is replaced with [REDACTED] before the diagnostic reaches the session. Credential references must be POSIX environment-variable names, resolved once per invocation.

One caveat lives inside audit.py. Most HTTP calls hand headers to curl through --config - (stdin), the documented reason the key stays out of curl’s argv — but two transport paths, curl_raw_request (raw requests, including the error-leakage probes) and stream_call (the Step 10 stream-integrity probe), build curl command lines with -H "x-api-key: …". During those steps the key is in the curl process’s argument list, readable by other local processes that can see the process table. The adapter side is careful; this part is in the audited script. No telemetry was found either: audit.py reports nothing back to the author, and its only outbound hosts are the relay URL you pass and documentation links.

Requirements

• Python 3 on PATH (python3, then python, then py -3 on Windows), launched through the harness subprocess service.
• curl — the standalone audit.py uses the Python standard library plus the curl executable for HTTP transport; no pip install is needed for it.
• Node ^22.19.0 || >=24.0.0, from the package’s engines field.
• Harness — the repository states its DSH compatibility in docs/skill-distribution.md, linked from the README as the DSH plugin’s distribution notes: “tested with DSH 0.1.0-rc.6 and dsh-cc-tui 0.4.1”. package.json has no dsh.compatibility block, and the main README itself names only “DSH Web and community TUI surfaces that use the official @deepseek-ai/dsh-commands registry”, without pinning a harness version.
• Peer dependencies: @deepseek-ai/cordis ^4.0.1, plus dsh-commands, dsh-credentials, dsh-llm, dsh-settings and dsh-subprocess, pinned at 0.1.0-rc.6 — install requirements, not a compatibility statement.
• Install-time scripts: none; scripts holds only test:dsh. The cc-tui profile’s own dsh-cc-tui@0.4.1 dependency needed the koffi@3.1.5 install script allowed in that profile’s pnpm build policy — a client requirement, not this bundle’s.
• Telemetry: not documented, and none found in the audit script. The README states no permission model; read its network behaviour as “your key goes to the relay URL you supply, and nowhere else”.

Known limits

• A run costs real tokens and a full run is not cheap; --connectivity only says whether the relay answers on either protocol.
• Findings are signals, not proof — the repository marks the identity, latency and channel detectors as needing corroboration, and inconclusive steps stay inconclusive rather than becoming safety verdicts.
• Non-Claude DSH routes are refused by design, because the identity and stream baselines are Claude-specific.
• The two curl paths above place the key in curl’s argument list for the duration of those requests.
• The repository’s verification document exercised the connectivity command through a minimal DSH service harness rather than either UI — it proves the adapter, subprocess composition, two protocol paths and the credential boundary, and explicitly not every interactive rendering path.

License, honestly

The license is AGPL-3.0-only: GitHub metadata says AGPL-3.0, package.json says AGPL-3.0-only, and the README agrees, so there is no divergence to flag. AGPL is strong copyleft with a network clause — modify this code and let users interact with it over a network, and you are expected to offer them the corresponding source; running it locally is not distribution and triggers nothing. The adapter and the generated audit.py ship under that license, and the README calls the choice deliberate, to keep modified network-service deployments accountable to the same source-availability standard the project audits.

When to use it — and when not to

Use it before you trust a relay with traffic you care about: a coding-agent workflow, package-install suggestions the agent will act on, or anything wallet-adjacent. Local execution is the point — the key goes to the relay you chose rather than a third-party testing site, and the report stays in your workspace. --connectivity is a reasonable first smoke test for a relay URL you have just been handed.

Do not install it expecting continuous protection: it runs when you invoke it, audits one target per invocation, and does nothing the rest of the time. Do not read LOW RISK as a clean bill of health — the README says it does not certify a relay as safe. And do not run the full audit casually against a metered relay; use --fast-context or skip flags for a cheaper pass.

FAQ

Does running the audit cost me money?

Yes: the no-argument default runs the full audit and “may consume metered tokens”. --connectivity is the cheap mode, with two max_tokens: 8 requests. The context-length step is the largest driver, and --aggressive-error-probes adds a 256 KB probe whose help text warns about metered billing.

Does it change anything on the relay, or in my project?

It does not modify the relay — POSTs to chat endpoints and a few informational GETs, with no PUT, PATCH or DELETE. Locally it writes the Markdown report and, only on request, an append-only JSON-lines transparent log. Under the DSH plugin both paths must stay inside the current session workspace.

Where does my API key go?

To the relay you choose, as the authentication header of the probe requests. The adapter resolves the named credential from DSH Credentials and passes the value to the Python process in API_RELAY_AUDIT_KEY, never as a command argument, with command input recording disabled; failure output is redacted before it reaches the session. Caveat: two transport paths inside audit.py build curl command lines with the key in a -H header.

Which harness versions are supported?

The repository’s DSH distribution notes state a compatibility contract of “tested with DSH 0.1.0-rc.6 and dsh-cc-tui 0.4.1”. The main README stops at naming DSH Web and “community TUI surfaces that use the official @deepseek-ai/dsh-commands registry”, and package.json has no dsh.compatibility block — its 0.1.0-rc.6 peer pins are install requirements, not a compatibility promise.

Can I audit a non-Claude relay with it?

Not through the plugin’s defaults: the adapter refuses to run unless the resolved model id — or the directory name and description for that model — contains “claude”. The relay’s own API may be Anthropic- or OpenAI-compatible; what is Claude-specific is the audit baseline. You can pass --model naming a Claude model, but the identity and stream-integrity checks were built for Claude-shaped responses.

What does an inconclusive result mean?

That the tool could not decide. A blocked probe, an unsupported format or an ambiguous response is not treated as safe, stays visible in the report with the rest of the per-step evidence, and never becomes clean. Read the per-step sections rather than only the final LOW/MEDIUM/HIGH line.