Designing Compact Neural Architectures via Neuron Gating and Mixed Activation

Neural Architecture Search is powerful but expensive due to discrete, combinatorial design choices. This paper proposes continuous relaxations of neuron-level and activation-level decisions, enabling fully differentiable optimization across MLPs, CNNs, RNNs, and Transformers. Three resulting methods (NAS-NG, NAS-MA, NAS-NGMA) find highly compact architectures --- including a CNN with just 0.26M parameters hitting 99.63% MNIST accuracy --- while outperforming standard DARTS on CIFAR-10. This offers a scalable, general-purpose toolkit for automatically designing efficient models, valuable for deploying AI on resource-constrained devices like mobile phones or edge hardware. Authors: Abhishek Shukla, Ankur Sinha, Faiz Hamid Paper: https://arxiv.org/abs/2608.14443v1