GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

LLMs have typically been evaluated on geo-related tasks in narrow, homogeneous settings, obscuring how well they generalize across diverse geospatial and temporal challenges. GeoBenchLLM addresses this by combining twelve public datasets into a comprehensive benchmark covering varied geo-related tasks and domains. This is useful for researchers and developers building geospatial AI applications—such as mapping tools, location-based services, climate or urban analytics, and geographic question-answering systems—needing to understand which model characteristics (the paper highlights reasoning ability and model size) most influence performance, guiding model selection for real-world geospatial deployment. Paper: https://arxiv.org/abs/2608.07411