TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

Vision-language models like CLIP have become foundational infrastructure for image search, multimodal AI assistants, and content moderation. Yet a persistent frustration is that image embeddings encode far more information than any caption captures, creating a mismatch that degrades retrieval and reasoning. TEVI uses captions as a scalpel rather than a label, selectively suppressing irrelevant image content to bring representations into closer alignment with what language actually describes. This has immediate applications in fine-grained image retrieval, cross-modal search, and any system where precise semantic matching between images and text matters — from e-commerce product search to medical image-report alignment. Authors: Sweta Mahajan, Sukrut Rao, Jiahao Xie, Alexander Koller, Bernt Schiele Paper: https://arxiv.org/abs/2606.07451v1