HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

Long-context LLM inference is bottlenecked by ever-growing KV cache memory during decoding. HiKV introduces an algorithm-hardware co-design that compresses the cache hierarchically: first evicting unimportant tokens within a budget, then loading only significant elements of retained tokens, achieving compression beyond single-granularity methods. A dedicated accelerator with a reconfigurable importance sorter unifies both stages in one circuit. Evaluated on representative LLMs, HiKV delivers up to 7.95x speedup and 90% energy reduction with negligible accuracy loss. Applications include efficient long-context LLM serving infrastructure, edge and datacenter inference acceleration, and hardware-software co-design for scalable AI deployment. Authors: Chao Fang, Jun Yin, Man Shi, Marian Verhelst Paper: https://arxiv.org/abs/2607.22389v1