NESC: Robust Neural End-2-End Speech Coding with GANs.

Abstract: Neural networks have proven to be a formidable tool to tackle the problem of speech coding at very low bit rates. However, the design of a robust neural coder that can be operated robustly under real-world conditions remains a major challenge. Therefore, we present Neural End-2-End Speech Codec (NESC) a robust, scalable end-to-end neural speech codec for high-quality wideband speech coding at 3 kbps. The encoder uses a new architecture configuration, which relies on our proposed DualPathConvRNN layer, while the decoder architecture is based on our previous work SSMGAN. Our subjective listening tests show that NESC is particularly robust to unseen conditions and signal perturbations.

Preprint: Arxiv, accepted at INTERSPEECH 2022

For this demo:

• NESC (our proposed model) is trained on the LibriTTS Dataset provided under Creative Commons License.
• SSMGAN is trained on the full training set of the VCTK corpus provided under Creative Commons License.
• All test items are unseen for both NESC and SSMGAN, and come from the CMU ARCTIC Dataset provided under the "free-software" License (page 13).
• The test items in the Section "CMU Speakers - noisy speech" are augmented using noise samples from the MS DNS-Challenge provided under Creative Commons License and room impulse responses from the Open SLR 28 Dataset provided under Apache 2.0 License.

Conditions of Use.

---

CMU Speakers - noisy speech

Original	NESC 3kbit/s (ours)	SSMGAN 1.6kbit/s	EVS 5.9kbit/s

---

CMU Speakers - clean speech

Original	NESC 3kbit/s (ours)	SSMGAN 1.6kbit/s	EVS 5.9kbit/s

---

Conditions of Use:

• Fraunhofer IIS generated this sound material based on material that is freely available on The Language Technologies Institute at Carnegie Mellon University for CMU ARCTIC, MS DNS-Challenge for background noise, and Open SLR 28 Dataset for room impulse responses.
• The content has been processed using generally accepted rules of technology as well as scientific care, but not actual attainment of any expected feature.
• With the exception of willful intent or gross negligence, Fraunhofer IIS shall not be liable that Open Source software or other third party software is free from any error or claim or its fitness for a particular purpose, even if included within the Sound Material.
• The Sound Material shall only be used for testing and appreciating speech synthesis techniques and shall not be copied, publicly transmitted, distributed, lent and modified for any other reason.
• No representation or warranties are made or implied regarding the accuracy, non-infringement, fitness for a particular purpose of Sound Material.
• Unless the conditions shown above are agreed by the recipient, no permission is granted for any use including copy, public transmission, distribution, lend, and modification.
• Copyright and Permission notice shall be duplicated whenever Sound Material are copied, distributed or publicly transmitted.
• The Sound material cannot be distributed with charge. ^[return]