Modern AI applications are rarely a single model call. A question-answering system chains a retriever and a reasoner; a data-processing pipeline runs schema inference, cleaning, and mapping; a coding assistant loops between a coder and a verifier. Each of these agents needs an LLM assigned to it — and with 23 candidate models and a handful of agents, the number of possible configurations explodes into the millions.

The problem

The two obvious strategies both hurt. Assigning the most expensive model everywhere (the high-cost extreme) gives high quality at high cost; assigning the cheapest everywhere saves money but degrades quality. What we usually want is in between: minimize cost subject to a quality constraint — keep quality within a small tolerance ε of what the high-cost extreme achieves.

Why existing methods overspend

Bayesian optimization and bandit methods can search this space, but they share one wasteful habit: to judge a candidate configuration, they evaluate it on the entire query workload, shrinking their cost/quality estimate all the way down to a point. That precision is often unnecessary — if a configuration is clearly below the quality threshold after a few queries, the remaining evaluations are money burned.

The SCOPE idea

SCOPE decides not only which configuration to try next, but also which query to evaluate it on. It maintains per-query cost and quality estimates, aggregates them into workload-level confidence ranges, and greedily evaluates the query that shrinks the workload-level uncertainty fastest. Two stopping rules end an evaluation early:

1. the quality estimate falls below the threshold (the configuration is doomed — reject it without finishing), or
2. the cost estimate is dominated by the current incumbent (it can't win — move on).

Guarantees and results

SCOPE's output satisfies the quality constraint with high probability, and its cost gap to the best feasible configuration goes to zero as trials increase — even under noisy LLM evaluations. Across three systems (Text-to-SQL, data transformation, data imputation) with 23 candidate LLMs, SCOPE finds cheap feasible configurations far faster than cEI, CONFIG, LLAMBO, Abacus, LLMSelector, and SafeOpt. On unseen workloads, the selected configurations cut cost by 95–98% versus the high-cost extreme while even improving quality by 5–21%.