CLIProxyAPI benchmark · Boundary and attribution

Start with CLIProxyAPI

Separate proxy capability from what the sidecar, harness, and UI add to the path.

Primary local sources: BENCHMARK-MEASUREMENT-PLAN.md, model-context-matrix.csv, and planned-run-manifest.json.
BENCHMARK-MEASUREMENT-PLAN.md · model-context-matrix.csv · planned-run-manifest.json

The idea in plain language

The product under test is CLIProxyAPI at 127.0.0.1:8317. Direct Chat Completions, Responses, and Messages measure the proxy. Warp, Codex CLI, and Claude Code are paired harness lanes. If a UI does not show a response that already crossed the network, that does not become a proxy failure.

Running example: send the same sealed parcel — nonce, checksum, and three needles — through all six lanes. If the harness changes the wrapping, you are measuring the lane too. The analogy breaks because context, tools, and thinking carry semantics, not size alone.

Attribution uses the smallest boundary where the failure reproduces: provider → proxy → compatibility layer → harness → UI. A direct and harness failure with the same error points to the provider or CLIProxyAPI; direct passing while the sidecar fails points to compatibility; failure before transmission belongs to the harness envelope.

Verifiable excerpt

primary_boundary = CLIPROXYAPI_DIRECT_127.0.0.1_8317\nproduct_under_test = CLIProxyAPI routing + translation\nharnesses_are_separate_lanes = true

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/model-context-matrix.csv · outputs/planned-run-manifest.json

The path in one picture

Running example: the same nonce + checksum crosses every lane; the failure belongs to the smallest boundary where it reproduces.

Retrieval check

Which boundary gets the primary ranking?