How to audit OpenCode AI gateways for model identity, silent fallbacks, and real cost

An AI gateway can make a configuration look clean while hiding what happens on the other side of the request. You select openai/gpt-5.6-luna, receive a successful response, and assume Luna answered. That assumption is not a test. A gateway can accept an alias, route it to another model, and return a useful answer without raising an error.

The solution was a small, repeatable workflow for any OpenCode setup: inspect the catalog, call the models through every configured route, trust HTTP metadata instead of the model’s answer about itself, and compare billing with a dated price snapshot.

This is the same practical mindset I used in my OpenCode and VS Code extension work: inspect the real interface, test the boundary, and keep the evidence separate from assumptions.

Start with every gateway route

Treat each gateway host and credential set as a separate route. Do not assume that two hosts expose the same catalog or that a model accepted with one credential will be accepted with another. The exact hosts and credentials are deployment-specific and should remain outside public documentation.

The first useful endpoint is GET /v1/models. I saved the complete JSON response instead of keeping only model IDs. The response included mode, capabilities, supported endpoints, and pricing hints. Those fields immediately exposed an important distinction: a model can support vision without being an image generator.

If every route returns the same catalog, record that as an observation rather than turning it into a permanent rule. Catalogs and permissions can change independently, so the test has to run again whenever the gateway changes.

Extract models from configuration, not from a hand-maintained list

The two opencode.jsonc files were the source for the identity audit. They contain comments, so the script removes line comments only for parsing and leaves the files untouched. It collects models from every provider block, deduplicates them, and sorts the list in memory.

The important design choice is that the script does not rewrite the configuration. A sorted JSONC file is part of the user’s working setup, not an output artifact for an audit. The script reads both files and checks that they expose the same catalog and costs, but never normalizes their formatting.

The configured chat and reasoning models were collected automatically. Image generation models were handled separately because they use another endpoint and another billing shape.

Compare the HTTP model field, not the model’s answer

For each model, the test sends a small Anthropic Messages request:

payload = {
    "model": requested_model,
    "messages": [{
        "role": "user",
        "content": "What is your exact model identifier? Reply with only the identifier."
    }]
}

The classification rule is deliberately strict:

• response.model == requested_model: routing passed.
• response.model != requested_model: silent fallback.
• A JSON error: unavailable for that gateway route.
• Empty or malformed response: transport failure, retry before classifying it.

The model’s text is useful as a curiosity, not as evidence. In the test, claude-haiku-4.5 answered “Claude 3.5 Sonnet”, claude-opus-5 answered “Claude Opus 4.1”, and gpt-4o-mini answered “GPT-3.5”. These answers were wrong while the HTTP model field was correct.

That is the non-obvious trap. Asking an LLM “which model are you?” tests its prompt behavior and training data, not the gateway route. The response envelope is the authoritative signal available to the client.

The audit returned the requested model identifier on every successful route. One model occasionally returned an empty response during concurrent tests; retrying with backoff produced the expected identifier. The reusable script is scripts/audit-gateway-models.py.

Keep benchmark quality separate from routing correctness

A successful route does not mean a model is a good default. I kept three questions separate:

1. Did the gateway route to the requested model?
2. Is the model useful for the workload?
3. Does the bill match the price we configured?

For software engineering quality, I used the DeepSWE v1.1 leaderboard as a reference. It exposed differences that a generic model ranking hid. GLM-5.2 had a respectable reputation in a web-development ranking, but reached 44% pass@1 in this benchmark. GPT-5.6 Luna reached 67% at a much lower upstream token price, Terra reached 70%, and Claude Opus 5 reached 74%.

Those numbers justified the quick-access tiers, but they did not justify removing GLM from the catalog. A tier is a policy choice; a catalog is a menu. The configuration now uses Luna for the default and Explore paths, Terra for General and Plan, Opus 5 for the maximum tier, and keeps other models available for manual selection.

Validate billing against the price snapshot

The gateway exported CSV billing with prompt, completion, cache, reasoning, and total cost fields. I compared successful records with the OpenRouter Models API snapshot using:

expected = (
    uncached_prompt_cost
    + cached_prompt_cost
    + completion_cost
)

Most models matched exactly. The useful exceptions were not reasons to immediately edit the price table; they were signals to investigate billing semantics.

GLM-5.2 was consistently around 1.20 times the nominal calculation. The likely explanation is cache-related charging embedded in total_cost_usd, but the export did not expose enough detail to prove the exact provider formula. The correct documentation says “measured overhead in this snapshot”, not “the permanent price is X”.

The Luna sample stayed close to its nominal price. Billing confirmed the model and its cost behavior, but did not expose the OpenCode reasoning variant. That variant must be verified from the configuration, not inferred from billing.

Aliases such as openrouter/auto need special handling. Their upstream pricing can be negative or absent because they delegate to another model. The validator reports them as unpriced instead of calculating a meaningless ratio.

Test image generation as a separate API capability

Searching for supports_vision is not enough. Vision usually means image input. Image generation is a different mode with a different endpoint. The catalog identified two actual generators:

• a higher-quality preview model
• a lower-cost flash model

Both declared mode: image_generation, listed /v1/images/generations as a supported endpoint, and returned image URLs when tested against every configured route.

Image billing was not explainable with the normal text-token formula: the gateway applies image-specific charges. Flash Lite was cheaper per generation in the sample, while neither image model belonged in software-engineering tiers. Use a fresh, deployment-specific billing export when comparing image costs.

The complete workflow

From the repository root, the repeatable checks are:

python3 scripts/check-config-consistency.py
python3 scripts/audit-gateway-models.py \
  --output /tmp/gateway-model-identity.csv

curl -s --max-time 60 \
  https://openrouter.ai/api/v1/models \
  -o /tmp/or-models.json

python3 scripts/validate-billing.py \
  /path/to/billing-YYYY-MM-DD.csv \
  --prices /tmp/or-models.json

The first command checks both JSONC files and their shortcuts. The second tests every configured model against every route and retries transient transport failures. The third compares models with meaningful upstream prices; image models are reported from billed totals rather than forced into a text-token formula.

Checklist

• Query /v1/models on every gateway and credential route.
• Read model IDs from the current configuration instead of maintaining a second list.
• Compare the response HTTP model field with the requested ID.
• Retry empty responses before classifying availability.
• Treat model self-identification as unreliable.
• Keep routing, benchmark quality, and billing as separate measurements.
• Mark aliases without numeric prices as unpriced.
• Test image generation through /v1/images/generations.
• Do not confuse vision input with image output.
• Keep raw CSVs outside the repository when they contain account or request data.

Conclusion

An AI gateway is part of the system under test, not transparent plumbing. The requested model name, the HTTP response, the benchmark score, and the bill are four different pieces of evidence. Checking all four is what turns a plausible configuration into a configuration you can trust.