ABR’s ASR models fill a critical niche on the Hugging Face ASR Leaderboard, achieving high accuracy levels with parameter counts well below alternatives. Our abr-ai/niagara-19m-batch.en and abr-ai/niagara-38m-batch.en models transcribe speech as accurately as competitors many times their size. We published them on Hugging Face to let everyone evaluate our models and compare them to the many popular open-source options out there.
But a model is only one part of a voice-enabled product. Speech happens in real time, and natural interaction is fast. Voice-enabled products must transcribe speech from the moment a user begins talking and deliver spoken responses promptly. This has to happen regardless of network quality. Many products have domain-specific words that aren’t in standard model training data sets, which still must be transcribed and spoken accurately. And synthesized speech responses that always sound the same feel robotic, so being able to vary the speed, volume, and pitch of spoken words is critical.
The ABR SDK is how we empower developers to build voice-enabled products with these kinds of features using our real-time speech AI models. Since these models are so compact, running voice-enabled products on all devices, including tiny low-powered ones, is easy. And on higher-end edge processors, running these compact models efficiently with the SDK leaves plenty of resources for other critical processing tasks.
The batch models provide an easy reference point
The batch models give developers a direct way to compare model performance with other open models on benchmark datasets and your own audio. niagara-19m-batch.en and niagara-38m-batch.en are production-grade batch ASR models. For applications that only need to turn recordings into text, they may be all you need.
The problem with batch models is in the name: they process audio in batches and produce transcriptions for that batch. Developers making their first voice-enabled products quickly realize that voice interfaces don’t neatly fit into equally-sized batches. Ideally, you would have a batch for each sentence, but how do you know when someone’s finished their sentence if you haven’t transcribed anything yet? You could make very small batches, but then you quickly realize that batch models need all the information in a big batch to correctly transcribe words at the start. The many different approaches to making batch models fit in a real time context never feel quite right.
That’s why our main focus is live streaming models. Most of our batch models have live equivalents, and we offer live streaming models for speech recognition and synthesis (text-to-speech) in a growing number of languages. These live streaming models run on many different hardware platforms through the ABR SDK.
The ABR SDK
The first ABR SDK release includes a Python binding library that communicates to model backends over a stable C ABI, with C and Java bindings releasing soon. The SDK supports Niagara ASR and Nith TTS models running simultaneously and can be deployed on Linux x86-64, Linux ARM64, and Android ARM64 platforms, with optimizations for edge NPUs and DSPs from leading silicon providers under development.
To keep things simple, each model is encapsulated in an application package that contains a compiled .so, the model weights, and any other supporting files used by the model. Switching to a different language or upgrading to newly trained weights only requires changing the path to the extracted application package. The C ABI is designed to be a minimal and flexible set of primitives that changes very rarely, meaning that most model updates do not require changes to language bindings. And most optimizations, like running on an NPU, are handled in the compiled backends, meaning that the language bindings are all that developers need to learn.
Keeping things simple does not mean that we have a limited feature set. Our stock ASR and TTS models can be augmented with custom vocabularies to recognize and synthesize domain-specific words. ASR transcripts go through a fully customizable post-processing pipeline including spell-checking and punctuation. TTS models understand the prosody tags from speech synthesis markup language (SSML), enabling speech with varied speed, pitch, and volume. All of this happens directly on the device, meaning that no internet connection is required for running models nor any of these additional features.
On-device speech AI example
The main data processing loop in a typical voice-enabled product performs does the following common steps: take in microphone data, perform speech recognition, do something with recognized speech to produce a response, text-to-speech synthesis, pipe audio out through a speaker. The SDK handles the speech recognition and text-to-speech synthesis parts, and lets you fill in what you want for the other parts. The internal logic could be anything; you can pass the transcript along unchanged, use it to set robotic motor control targets, or even send it to a cloud LLM to enable real conversations.
The example below is real functioning Python code that does everything except the business logic in less than twenty lines, including imports.
import sounddevice as sd
from abr_sdk.asr import Asr, AsrTranscript
from abr_sdk.tts import Tts
transcript = AsrTranscript()
with Asr(ASR_LIBRARY_PATH) as asr:
with sd.RawInputStream(samplerate=16000, channels=1,
dtype="int16") as stream:
while listening: # stop condition
data, _ = stream.read(1600) # 100 ms of audio
asr.push(bytes(data), on_chunk=transcript.chunks.append)
asr.wait_for_completion()
with Tts(TTS_LIBRARY_PATH) as tts:
with sd.RawOutputStream(samplerate=16000, channels=1,
dtype="int16") as speaker:
tts.push(transcript.text.encode("utf-8"), on_pcm=speaker.write)
tts.wait_for_completion()
On the input side, asr.push(chunk, on_chunk=…) processes audio as fast as the microphone produces it, no need for batching. The model runs on a separate thread and produces AsrChunks when speech audio is processed. How you generate a response to the transcript is up to you, from simple semantic or intent matching, to using a full-blown LLM. Then on the output side, tts.push(text, on_pcm=…) produces audio data almost instantaneously, resulting in a voice interface that feels completely natural.
In addition to being simple, fast, and deployable on small edge devices, a voice interface like the above keeps voice data on the device, protecting your user’s privacy. And if you choose to send that transcript to the cloud for processing, you now only have to send and receive bytes of text instead of kilobytes of audio data.
All language variants of niagara-38m-live have 38 million parameters and produce first text from first speech audio in as little as 115 milliseconds3. nith-5m-live is a 5.5 million parameter model and produces first audio in as little as 147 milliseconds3. Regardless of the platform, our models always run faster than real-time, meaning an RTF below 1.0. Our first post walks through the state space architecture that lets models reach this performance with so few parameters.
Getting started
We recommend you start trying out niagara-38m-batch.en on Hugging Face, which includes a live demo space where you can see how our small batch models handle your audio. The next step is to sign up on our developer portal, and follow the SDK documentation to install the SDK and use our application packages. Try running streaming recognition and synthesis on your hardware and confirm for yourself how fast and natural voice interactions can be. If you want to go even further and deploy your voice-enabled product commercially, we would love to speak with you.