LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning

When companies need to remove sensitive personal data memorized by an LLM, current "unlearning" techniques are typically judged only by whether the model stops outputting the information—not whether the underlying knowledge is actually erased from its parameters. This creates risk, since obfuscated knowledge can often be recovered through resurfacing attacks. LACUNA addresses this by embedding synthetic PII into known parameter locations of OLMo models, allowing researchers to directly verify whether unlearning methods target the correct weights. This testbed is valuable for privacy compliance, GDPR-style "right to be forgotten" requirements, and building genuinely trustworthy data-removal tools for deployed models. Authors: Matteo Boglioni, Thibault Rousset, Siva Reddy, Marius Mosbach, Verna Dankers Paper: https://arxiv.org/abs/2607.02513v1