CLIProxyAPI benchmark · Functional behavior

Test tools and thinking by contract

A valid tool call and a visible reasoning block answer different questions.

Primary local sources: BENCHMARK-MEASUREMENT-PLAN.md, model-context-matrix.csv, and planned-run-manifest.json.
BENCHMARK-MEASUREMENT-PLAN.md · model-context-matrix.csv · planned-run-manifest.json

The idea in plain language

The canonical profile exposes only get_benchmark_nonce; the real profile preserves the harness envelope. Nothing executes shell, network, or writes. Thinking receives an observable class, and models compare only when class and effort match.

Running example: send the same sealed parcel — nonce, checksum, and three needles — through all six lanes. If the harness changes the wrapping, you are measuring the lane too. The analogy breaks because context, tools, and thinking carry semantics, not size alone.

Tools require a name, valid JSON, inert local execution, call two, and correct use of the result. Thinking may be NOT_EXPOSED, HIDDEN_BUT_USAGE_REPORTED, SEPARATE_STREAM, VISIBLE_IN_CONTENT, SUMMARY_ONLY, or NORMALIZED_BY_COMPAT_LAYER. Hidden thinking does not prove no internal reasoning.

Verifiable excerpt

tool = get_benchmark_nonce · side_effects = none\nthinking_class ∈ {NOT_EXPOSED, SEPARATE_STREAM, SUMMARY_ONLY, ...}\ncompare_only(equal_class && equal_effort)

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/model-context-matrix.csv · outputs/planned-run-manifest.json

The path in one picture

Running example: the same nonce + checksum crosses every lane; the failure belongs to the smallest boundary where it reproduces.

Retrieval check

When may two models compare thinking?