Online Pandora's Box for Contextual LLM Cascading

Running multiple AI models and deciding which to query, in what order, and when to stop is an increasingly common engineering challenge. Calling a powerful but expensive model for every query is wasteful; calling a weak model for hard problems is costly in accuracy. This paper formalizes that tradeoff through elegant economic theory, treating each API call as opening a box whose value is uncertain until revealed. The result is a principled, adaptive policy that learns optimal querying strategies from experience. Practical applications span cost-efficient AI infrastructure at scale, multi-provider routing systems, and any organization managing a portfolio of AI models with heterogeneous cost and capability profiles. Authors: Alexandre Belloni, Yan Chen, Yehua Wei Paper: https://arxiv.org/abs/2606.07392v1