Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

Building a high-quality speech synthesis system typically requires training multiple specialized models independently, then orchestrating them at inference time — an expensive and memory-intensive process. This paper explores a more compact path: starting with a speech classifier already trained to recognize acoustic properties, and attaching a lightweight generative subnetwork that reuses its internal representations. The result is a single-backbone model capable of conditional speech generation, reducing both memory footprint and compute cost. This approach is especially attractive for on-device deployment scenarios — hearing aids, mobile assistants, edge robotics — where model size and inference cost are hard constraints.