The promise of edge AI is straightforward: machines will interact with people as agents in a natural and secure way — anywhere and any time. To fulfill this promise, AI capabilities need to be more broadly enabled on edge devices, reducing network dependency, cloud processing costs and end user data exposure. What’s holding us back is that the dominant model architecture — the transformer — was designed for data centers and not edge devices. Data center servers have abundant memory, kilowatts of power, and fast, always present network connections. Edge devices have none of these.
Deploying a transformer on edge devices is not simply an optimization problem that targets less capable hardware. Quantization helps. Pruning helps. But the core scaling issue — computation layers whose cost grows as O(N²) in sequence length — remains. You can make a transformer smaller, but its dominant self-attention layer still carries that quadratic cost on hardware where compute and power are precious.
State Space Models (SSMs) offer a fundamentally different approach. They are not a scaled-down version of transformers. They are a different class of sequence model — one that replaces the most expensive part of the transformer stack with something architecturally cheaper. That difference turns out to matter enormously once you move off the cloud.
What is a state space model?
SSMs have their roots in control theory, which is used to describe how dynamic physical systems — autopilots, sensors, mechanical processes — respond to inputs over time. The core idea is simple: instead of examining the entire history of inputs every time you need to make a decision, an SSM uses a dynamical system to maintain a compact internal “state” that summarizes what has happened so far. As new data arrives, the state immediately updates. Output depends on the current input and the current state — nothing more.
This idea found its way into deep learning through work in my lab at the University of Waterloo. In 2019, my collaborators Aaron Voelker, Ivana Kajić, and I introduced the Legendre Memory Unit (LMU) at NeurIPS — the first State Space Model in deep learning. Shortly thereafter, we demonstrated that LMU/SSM-based architectures could match transformer accuracy on language modeling tasks with ten times fewer training tokens. The models now generating the most excitement in the research community —Mamba, Jamba, Samba — are all built on that foundation.
The key property that makes SSMs work is that the recurrent state transition is a linear operation. This is not a minor implementation detail. It is why SSMs avoid the vanishing gradient problem when training at scale, that made earlier recurrent networks impractical to train: because the memory is linear, gradients flow cleanly across arbitrarily long sequences. The linear dynamics are also why SSMs are efficient at inference: they provide a fixed-size state that updates one step at a time, with constant memory regardless of how much history has accumulated.
The architectural fork
Transformers work by comparing every element of a sequence to every other element — the attention mechanism. This is powerful, but it comes at a cost: compute and memory both scale as O(N²) in sequence length. Double the context, quadruple the work.
SSMs have O(N) scaling. As sequence length grows, computational cost grows linearly. At inference, the memory footprint is constant — the model maintains a fixed-size state regardless of how much it has processed.
Think of it this way: a transformer is like reading the entire transcript of a meeting to answer a single question. An SSM is like a sharp assistant who has been keeping running notes and can answer instantly.
In practice, this is not a binary choice. In practice, the most effective architectures are hybrids — predominantly SSM layers that handle the bulk of sequence processing at linear cost, with a smaller number of attention layers where global context genuinely helps. This is true in the cloud and at the edge. ABR’s edge models use this hybrid approach — SSM-dominant designs where the ratio of SSM to attention layers is tuned for the target power and latency envelope. The more SSM layers in the mix, the closer the overall cost profile is to O(N). The right question is not “SSMs or transformers?” but “how many attention layers does your use case require?”
Why the difference matters at the edge
Power. SSM layers require fewer multiply-accumulate operations per inference step than attention layers processing the same sequence. In an SSM-based architecture, the bulk of the model runs at linear cost. Combined with smaller overall parameter counts, this translates directly to lower power draw on any hardware from leading manufacturers — CPU, MCU, GPU, NPU or other accelerators. The advantage compounds for always-on applications: a model that draws even 50mW less in continuous operation adds hours of battery life to a wearable and frees thermal and power headroom for other subsystems. A detailed power analysis across common hardware platforms is the subject of an upcoming post in this series.
Latency. SSMs do not need to buffer a full context window before producing output. ABR’s ASR streaming models can produce a first token 120 milliseconds after the first audio arrives. This is first-first latency: the time from when the user starts speaking to when the device starts responding.
Critically, most ASR vendors report last-first latency — the time from when the user finishes speaking to when text appears. That metric lets the model buffer the entire utterance before processing begins, which is useful for batch transcription but meaningless for interactive voice. If you are building a voice-first product, first-first latency is the number that determines whether the interaction feels instant or broken.
At one-second latency, ABR’s models have competitive on word error rate against alternatives that require six times the wait or more. At 120ms latency, no other published model operates at all. This is not an incremental improvement. It is a latency category that only SSMs can reach at this scale.
Model size. Parameter count is the most portable proxy for deployment cost. It determines RAM footprint, flash storage requirements, and — to a first approximation — inference compute. ABR’s 19M-parameter ASR model fits comfortably in the SRAM of a modern MCU or DSP. Despite similar accuracy, a 74M-parameter Whisper Base model often cannot, requiring external memory access that adds both latency and power draw. This is not a question of more clever quantization. It is a structural advantage: SSMs reach competitive accuracy at parameter counts that transformers cannot match.
We will publish more detailed benchmarks — including accuracy, power, latency, and real-time factor on common hardware platforms — in an upcoming post.
Where this matters
These constraints are not abstract. Consider two exemplary voice-first product categories where they converge.
Voice-centric robots require always-available voice interaction. A robot cannot afford cloud instabilities or delayed round-trips mid-conversation, cannot run a billion-parameter model on embedded hardware, and runs from a battery. High accuracy, low latency, low power, and small model size, are all non-negotiable — simultaneously.
AR glasses make voice the natural input modality when hands are occupied and a screen is in the field of view. Any lag between speech and response breaks immersion. Privacy is especially acute: a growing number of users are unwilling to accept continuous audio streaming to the cloud from a device they wear all day. And, current voice applications can drain the small battery in glasses in under an hour.
ABR’s on-device SSM based voice inference models perform within the constrained resources of voice-centric robots and AR glasses in a way that no other ASR solutions can. We also provide SSM-based text-to-speech (TTS) that brings similarly impressive capability to these products, which need to speak as well as to hear.
Looking even more broadly, the same architectural advantages extend beyond voice. SSMs are naturally suited to any continuous time-series application — biometric data monitoring, sensor fusion, anomaly detection. The architecture generalizes because the underlying problem is the same: processing sequential data efficiently on constrained hardware. We are actively developing solutions in several of these domains, and the results mirror what we see in voice — SSMs deliver state-of-the-art accuracy at a fraction of the compute budget of alternatives. We will share specifics about these types of signal processing applications, as well as TTS, in upcoming posts.
What this means
SSMs are not a research curiosity. They are in production today and running on standard hardware with accuracy levels that compete with transformer-based models twice their size. Applied Brain Research has been building SSM-based systems since we introduced the foundational architecture in 2019. If you are designing a product that needs to hear, understand, or respond — and needs to do it on-device — reach out. Let us bring ABR’s SSM advantage to your products.
The next wave of AI products will be defined by what runs without the cloud. State Space Models are the architecture that makes that possible.