AI

Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning

Researchers have developed a new framework for automatic speech recognition (ASR) that can handle multiple languages and accents. The system combines a frozen speech encoder with a decoder-only language model to guide transcription. To improve performance, the researchers use contrastive learning to align speech and contextual representations in a shared embedding space. Experiments on over 1,500 hours of real-world conversational speech across 11 languages show that contextu
Researchers have developed a new framework for automatic speech recognition (ASR) that can handle multiple languages and accents. The system combines a frozen speech encoder with a decoder-only language model to guide transcription. To improve performance, the researchers use contrastive learning to align speech and contextual representations in a shared embedding space. Experiments on over 1,500 hours of real-world conversational speech across 11 languages show that contextual input improves recognition quality, with an overall performance gain of over 5% when contrastive alignment is applied. --- Why it matters: This work matters because it addresses two key challenges in ASR: limited multilingual support and the absence of principled alignment between speech and contextual representations. The framework's ability to handle diverse languages and accents makes it a significant step towards developing more robust and generalizable ASR systems. Source: https://arxiv.org/abs/2603.06505

This article was originally published at: https://arxiv.org/abs/2603.06505