CLIProxyAPI benchmark · Comparable measurement

Separate network from experience

TTFT, TTFE, TPS, and latency compare only when they start and end at the same boundary.

Primary local sources: BENCHMARK-MEASUREMENT-PLAN.md, model-context-matrix.csv, and planned-run-manifest.json.
BENCHMARK-MEASUREMENT-PLAN.md · model-context-matrix.csv · planned-run-manifest.json

The idea in plain language

On the network, the clock begins at t_send. In a harness, it begins at process start or submit. Network TTFT and harness TTFO are therefore different numbers. Client overhead is a paired difference, never automatic blame assigned to the proxy.

Running example: send the same sealed parcel — nonce, checksum, and three needles — through all six lanes. If the harness changes the wrapping, you are measuring the lane too. The analogy breaks because context, tools, and thinking carry semantics, not size alone.

Network TTFT = t_first_visible − t_send; TTFE = t_first_event − t_send; visible TPS = visible tokens ÷ visible interval; network total latency = t_done − t_send. Harness TTFO and total latency use process start/exit. Non-streaming gets no TTFT.

Verifiable excerpt

network_TTFT = t_first_visible - t_send\nharness_TTFO = t_harness_first_output - t_process_start\noverhead_harness = E2E_harness - network_at_proxy_input

outputs/BENCHMARK-MEASUREMENT-PLAN.md · outputs/model-context-matrix.csv · outputs/planned-run-manifest.json

The path in one picture

Running example: the same nonce + checksum crosses every lane; the failure belongs to the smallest boundary where it reproduces.

Retrieval check

How is harness overhead reported?