AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

As LLMs take on roles beyond code generation, acting as autonomous scientific agents, we need ways to test whether they can actually run experiments intelligently. AgentHPOBench evaluates this directly by having agents sequentially tune machine learning hyperparameters, observing results and logs before deciding on the next configuration. Testing 12 agents across 30 tasks, the benchmark exposes real limitations in iterative reasoning and log interpretation. This work matters for anyone building AI research assistants or AutoML systems, showing where current agents still fall short of matching human researchers\' experimental judgment and persistence. Authors: Tianyu Huai, Tingshuo Fan, Xinchi Chen, Yining Zheng, Yuxin Paper: https://arxiv.org/abs/2607.29626v1