There is only one way most teams know how to make a small model: start with a big one and take things away. Distill it, prune it, quantize it, and keep going until it fits on the device. It is a reasonable instinct, and it produces reasonable results. It also means that every design decision inside that model was made for a machine with a kilowatt of power and gigabytes of memory, and none of those decisions get revisited on the way down.
At ABR, we went the other direction, and we did it in two ways at once.
The first is starting small. Our first English speech recognition model was 8 million parameters. We picked that number by looking at the memory and the power budget of an edge processor, before we had set any accuracy target at all. The constraint was the input to the design.
The second is thinking big. The architecture we put inside that constraint was not something we found by experiment. We invented a new architecture (State Space Models; SSMs). As a consequence, the new Niagara models are a hybrid, with SSM layers carrying the temporal representation and attention working alongside them. We derived the SSM to solve a specific engineering problem: running in continuous time with continuous input. It was in the design from the first 8 million parameter model onward.
The 8M model cleared 85% accuracy. We then grew the same architecture upward, and at 38 million parameters it passed 90% on a public third-party leaderboard, sitting within a third of a percentage point of a model six times bigger.
I want to explain why those two strategies together produce something that neither one produces alone.
Starting small: the ceiling came first
When we opened the Niagara roadmap, the requirement was a parameter ceiling below 10 million. This was not a target. It was a must. The model had to fit in the memory and the power envelope of a resource constrained edge SoC, and accuracy had to be raised until it met that constraint. We did not relax the constraint to meet an accuracy number.
The 8M model reached 85%. That told us the architecture worked at this low limit, which is the hard part. From there the roadmap grew upward: 9 million, 19 million, 38 million. Every step added capacity to an architecture that had already been proven inside a tight envelope.
This matters more than it might sound. A model that has been compressed down from a large one inherits the architecture and assumptions of its predecessor, and it inherits how that architecture uses memory and structures input. The parameter count comes down. The assumptions of the computation stays the same. You end up with a small model that still behaves like a large one, and on an edge compute platform that is exactly what costs you.
Direction of travel decides what gets optimized. Starting at 8 million forces state dimensionality, vocabulary size, operator selection and quantization to be settled against a hardware budget while the architecture is still being designed. Starting at a billion settles all of those against an accuracy curve and revisits them afterwards, when these important choices have influenced the result.
The same discipline gave us Nith, our text-to-speech family, at about 5 million parameters. It is worth remembering what the landscape looked like when we started designing it. The models we were measuring ourselves against were cloud systems above a billion parameters. The smaller reference points people cite today, at 99 million and more, arrived at roughly the same time we did.
Size was also the easy part of the Nith requirement. Producing a voice that sounds genuinely natural, that supports cloned voices, and that responds to prosody and emotion control cues, all at about 6 million parameters, was a serious piece of engineering. Niagara and Nith use different architectures. What they share is the method. Two architectures, one discipline, both landing an order of magnitude below the industry norm.
Thinking big: what the architecture contributes
Niagara is a model with a hybrid architecture. The SSM layers carry the continuous temporal representation at constant cost per step. Attention does content-addressed comparison across the sequence, where that comparison genuinely helps. Neither one is there by accident.
The efficiency comes from the SSM layers doing critical temporal representation work that attention would otherwise have to do.
SSMs came into deep learning through my lab at the University of Waterloo, when Aaron Voelker, Ivana Kajic and I introduced the Legendre Memory Unit at NeurIPS in 2019. Our Legendre Memory Unit (LMU) was a derived answer to a specific question: how do you hold a window of recent signal history in a fixed-size state. In fact, it was derived in part by looking at how the brain solves this problem. Because it was derived rather than discovered by scaling experiments, the cost structure falls out of the mathematics instead of having to be measured after the fact.
As well, SSMs are recurrent networks, and two consequences follow. They are worth keeping apart because they have different impacts.
1) Constant state fixes the memory budget. The state size of a recurrent network doesn’t change with more input. It is known at design time and sits in a fixed memory allocation. It does not grow over the course of a session. This makes memory usage small and predictable.
Specifically, with context length N, a state space layer scales O(N) in compute and O(1) in memory. Attention scales O(N2) in compute and O(N) in memory, because the key-value cache grows with every token processed. A hybrid pays the second cost only in the places where it buys something.
2) The recurrent form gives incremental processing. The model consumes audio step by step, without needing a whole utterance batched before it can produce accurate output. It is natively streaming. This is a property of the recurrence, and it is what makes real-time voice feel like a conversation.
I want to be precise about one thing, because it gets overstated. Streaming-native does not mean zero look-ahead. In our live models the causal stream emits immediately, and the non-causal stream deliberately waits a short while before emitting text, trading a bounded amount of latency for increased accuracy. The problem with transformers is that incremental processing with a fixed-size state is unnatural: you train with long batches and deploy with short ones, significantly reducing your real-world performance.
This explains the numbers we publish. Roughly 170 milliseconds from first audio to first text (see below for details).
The mix of state space layers and attention is chosen against the parameter budget rather than inherited as a default ratio, and once chosen it is fixed. That composition is why Niagara sits at 38 million parameters instead of in the hundreds of millions.
What the leaderboard shows
Niagara-38m sits on the Hugging Face Open ASR Leaderboard at 90.64% accuracy. That is within 0.3 percentage points of whisper-small, which carries 244 million parameters. It is above whisper-base at roughly twice its parameter count, and nearly four points above whisper-tiny. Its inverse real-time factor (RTFx) is 2623, against 348 for whisper-base.
Niagara-19m is on the same leaderboard at 88.99%, above whisper-base at under a third of its size.
Here is the part I find most telling. Niagara is the only entrant on that leaderboard built on state space layers. Every other model there uses a transformer-based architecture. The architecture that the research community has been most excited about for the last three years is, in production speech recognition systems, still ours alone.
It is also worth saying plainly what the top of that leaderboard now looks like. The leading models are separated by less than one word error rate percentage point, and they sit between 1 and 3 billion parameters. At that scale, accuracy rank has stopped being the deciding variable for anyone building a product. Parameter efficiency, latency and deployability are what decide it, and those are the axes where we lead.
The nearest comparator at our scale is Moonshine, which publishes at 34, 123 and 245 million parameters using a sliding-window transformer encoder. It is good work, and it is aimed at the same problem we are. It is also the exception that proves how empty this part of the field still is.
Why our models land well on real silicon
There is a third reason we’re able to hit amazing numbers on hardware. And, it comes from our team’s expertise, rather than the mathematics.
Our engineering team has designed arithmetic units, memory hierarchies and compiler backends. That is an unusual background for a team building speech models, and it changes what happens when a model meets a real piece of hardware.
The target is always the lowest achievable latency on the silicon the customer has already chosen. On edge devices, that latency is governed largely by how efficiently data moves, so we build the deployment graph for memory throughput.
Operator choice is one aspect that serves that goal. The same computation can be expressed as a single batched operation or as a series of fully connected operations, and those two move data very differently. Which one performs better varies with the platform and its compiler. Separately, any operator the accelerator does not support falls back to a CPU implementation, and that fallback is paid on every single inference, so we build the graph from what the hardware actually supports.
As well, precision is assigned per layer across FP32, INT16 and INT8, chosen to maintain accuracy at the lowest latency and power the target hardware allows. Not every processor supports the same numeric formats, and the right assignment is different on each one.
None of this is a search. We are not sweeping configurations at compile time. Our team already knows how these processors behave, so the right formulation is selected directly for each one.
This deep hardware knowledge is worth a lot in practice. On one application processor, moving the eligible operators onto the on-board NPU took first-token latency from 200 milliseconds to 165, and dropped CPU utilization from 99% to 27%. That latency gain matters. Critically, the utilization figure matters even more broadly, because it is a compute resource handed back to every other subsystem on the device.
Across the full set of edge platforms we have measured, from MCU class parts with a small neural accelerator up to application processors, Niagara first-token latency runs from 118 to 250 milliseconds, averaging 172 across 5 different hardware platforms. All the models that we have measured across all hardware platforms run in real-time with RTF below (typically well below) 1.0, meaning they can keep up with real conversation in live applications. Not one of them needed a GPU.
What this adds up to
Our models are small because the resource constraints came first. They are accurate at that size because the architecture was chosen from theory rather than inherited from shrinking someone else’s. And they run well on the hardware our customers are already designing around because the people who built them have built silicon.
Starting small gave us the envelope. Thinking big gave us the efficiency inside it. The leaderboard shows the comparative advantage.
What our customers and hardware partners react to is something more visceral: watching an accurate model run in real time on their own silicon with immediate response. Every edge platform we have benchmarked runs Niagara at or faster than real time, and that includes microcontroller-based chips. That is consistently the result people find surprising, and it is the result that changes what they believe they can build.
If you are a silicon manufacturer evaluating model portfolios for your platform, an OEM engineering team looking at on-device speech, or a developer who wants to try this today, we would like to hear from you. Real-time streaming variants with sub-170ms latency are available under commercial license.
ABR’s open-source batch models on Hugging Face demonstrate the power of SSMs for efficient ASR. For real-time streaming, multilingual support, hardware-optimized deployment, and our full voice-based speech AI pipeline, visit appliedbrainresearch.com or contact us for commercial licensing.