18 Oct 2026 10 min read m5stick stt

Whistle Hungarian

There is a super interesting model that was released a while back, Cactus-Compute/whistle, which is a small speech-to-text model — a single 16.9 MB file claiming a WER as low as 4.31% (on LibriSpeech test-clean).

Since I love a challenge when it comes to cramming anything onto the small M5Stick devices I own, I took it on.

Model running on the stick

Size optimization is always like saving a hot air balloon from crashing: deciding what needs to be kept on the "ship" and what can be thrown out. And then just starting to throw things out. :D Cramming everything onto an M5StickS3 with an ESP32-S3 is always a challenge. 8 MB of flash and 8 MB of PSRAM are always tight.

The first challenge is that the C++ engine that runs Whistle on CPUs (libneedle) only ships as a compiled library. The JAX training code is open, the inference engine is not. Rebuilding things with AI from the outside seems to be this week's topic anyway, with someone recreating the Adobe Suite using AI. I won't publish the implementation, since keeping the engine closed was the authors' original intent. I will only publish compiled builds.

After the initial code was created, the partition table looked like this:

The packed EN model is 7,584,960 bytes, which needed some fiddling and throwing away a couple of useful things. Remember, the base model is 16.9 MB, but the stick has 8 MB of flash in total, and only about 7.3 MB of that is left for the model.

The engrams do help with decoding, but we don't have the space for them, and the decoder is actually trained to be cut down this way.

On my small test set (16 English sentences, 168 words, read by macOS text-to-speech), the official engine scored 8.9% WER with the full model, 6.5% at depth 2 (awesome, but on a set this small probably an outlier), and 9.5% at depth 2 without engrams. My reimplementation running the packed blob scored the same 9.5%, so the C port matches the official engine on the configuration that fits.

And running it on the stick turned out to be a mess — hitting the second common issue: CPU speed.

Speeding up the model

I had never used the S3's advanced "AI features", described in the official docs as:

AI Acceleration: Includes vector instructions and dedicated SIMD (Single Instruction, Multiple Data) capability, which significantly speeds up neural network computing and signal processing tasks

At this point, processing took 2.4× the audio duration, around 25 seconds to transcribe a ten-second input, without listening to anything else during that time. That is ... how to say it kindly ... not the best experience. I am not an expert in this field, but Opus 5.5 and GPT 6.1 Sol had some ideas, and I threw in my ideas as well. Teamwork in 2026, right? :D

I can explain my part, but since I like to understand AI code as well, I did go into the details of all of this. Learning the tricks always helps with future projects.

The changes were the following:

1. Silence gate: silent or steady-noise audio is rejected right after feature extraction (~0.5 s), before the encoder runs.

2. Doing less work

   • Output head: scoring all 6955 tokens with 4-bit weights at every step was expensive.
   • A 2-bit sketch, later 1-bit (signs only), shortlists 32 candidates, which get exact scores.
   • Search: beam 5 → 3 → greedy by default.
   • Normalizer: greedy decoding computes the log-sum-exp over the top 6 candidates instead of all logits.
   • Pause squeezing: pauses of 300 ms or more are shortened to 160 ms in the feature domain, giving about 20% fewer encoder frames on natural speech.
   • Small wins:
     • Text-token scans over a prefix.
     • Lookup-table sigmoid/exp.
     • Two real frames per FFT.
     • Angle-addition position embedding instead of sinf/cosf per element.

3. Memory: where data lives decides the speed

   • The hierarchy: internal SRAM is fast, but only ~30 KB is free. PSRAM bandwidth was ~56 MB/s. Flash reads come through the same cache.
   • Hot buffers in internal RAM, allocated first because that heap fragments. Later: avoiding slow RTC memory that the heap also labels "internal".
   • Decoder staging: decoder weights copied to PSRAM once per utterance (2.3× faster than flash), later with float norms precomputed.
   • Cache lines switched to 64 bytes at boot, at runtime. The prebuilt config uses 32, so PSRAM bandwidth went from 56 → 90 MB/s, reducing overall processing time by about 9%.
   • int16 encoder lanes (the 4-lane residual stream) halved that traffic, with no WER change.
   • Access patterns:
     • Cache-friendly loop orders.
     • Fused passes, so data is read once instead of 3–4 times.
     • Transposes done in tiles. The decoder memory's byte-per-line scatter went from 185 → 30 ms.

4. The SIMD unit (PIE)

   • int8 matmuls with Hadamard-rotated activations quantized in fixed point, so weights are never dequantized to float.
   • int8 attention (head-major K/V) and int16 Kronecker products for the HadamardMLP, with operands held in registers.
   • Bit-plane kernels: the 2-bit codebook [−127, −38, 38, 127] is exactly −127 + 89·b0 + 165·b1. A dot product therefore comes straight from shifting and masking the packed bits, with no table lookup and identical results. Same trick for the 1-bit head.
   • Row mode: one activation scale per frame, with group norms folded into the weights, so each output row is a single accumulator chain.
   • Measured on the chip, which shaped all of the above:
     • A 128-bit load costs 2 cycles.
     • Reading out the accumulator costs about 9 cycles.
     • So the kernels are designed around loads and readouts, not multiplies.

5. Two cores

   • wh_parallel everywhere: features, mHC passes, attention prep, activation quantization, decoder memory.
   • Split choice: splitting matmuls by output rows rather than frames, so the cores don't both decode the same weights.
   • Ceiling: two cores give only ~1.35× the speed, because they share the PSRAM bus.

6. Compiler and FPU lessons

   • O3 + IRAM for hot kernels. GCC 8 most likely miscompiled one fused kernel at O3 early on (garbage text until it was built at O2). After it was rewritten, O3 was fine.
   • Float compares and branches are slow: integer abs-max and the hardware rounding instruction (round.s) replaced them in the quantizers.
   • GCC 8 has no instruction scheduling for this core, and its multiply-add forces register copies. Hand-interleaved SiLU, norm and elementwise loops: SiLU went from 47 → 29 cycles per element.
   • esp-dsp's S3-optimized FFT for feature extraction.

And for anyone trying to reproduce this, these were the fails:

• DCache preload (no extra bandwidth).
• Branch-free sigmoid via float selects (slower).
• Dropping encoder layers (WER 49–100%).
• Hardware cache autoload (hangs the chip).
• DMA double-buffering.
• Turning off multiply-add contraction globally (slower).
• Interleaved row blocks between the cores.
• A smaller head sketch (costs WER).
• Streaming without look-ahead (WER 50–90%).
• An early-stop guard based on attention position.

As it turns out, the S3's advanced AI features are just a set of 128-bit vector (SIMD) instructions — up to 16 int8 multiply-accumulates per instruction, no NPU or matrix unit — which is sad, but still, they can help a lot. Interestingly enough, the easiest ideas (mine) yielded some of the largest wins, like not processing silence (duh), RAM and allocation magic I learned from past projects (ESP32-specific), and doing things in parallel (another you-don't-say).

With all of this, processing went from 2.4× the audio duration to about 0.85× — faster than real time, using both cores. (The two-core split was there from the very first version. It just never gave more than ~1.35×, because both cores share one PSRAM bus.)

My pet benchmark that I used for this (processing time on the stick, greedy decoding, the default):

I was a happy camper once we got these results. At this point, I just directed the LLMs and didn't really touch any code. I know, vibecoding, but the question was: can I squeeze the code onto the stick without butchering everything, while running in real time?

Let's teach Whistle some Hungarian

Hungarian is a hard language. Words are long and carry many suffixes, so a single wrong letter makes a whole word wrong, and WER punishes that hard. CER (character error rate) shows better how close a transcript really is, so it matters more than WER in some ways. Since the original code includes LoRA fine-tuning options for other languages, I started doing just that. I did not know which datasets to use, but there were some sources I could use (I bet there are others as well), plus the special sauce: my own lectures.

Nothing crazy here. After a couple of rounds, I did have a model. The training ran with the same 2-bit quantization and the same cut-down 2-layer decoder as the stick, so what is measured is what runs on the device (greedy decoding, 1000 test clips per set, WER / CER):

Since it took hours, I did cheat and take inspiration from Mimo's dashboard. Mine looks like this: Dashboard. An eye-watering €0.81 of GPU electricity was burned on the project (3.23 kWh at €0.25/kWh). :D

As I said, nothing special here, but now I do understand a bit better why everyone wants good-quality datasets. More epochs on the same data stopped helping after run E. Introducing each new dataset did yield improvements, so if someone has 4 million hours of hand-annotated Hungarian voice recordings, feel free to reach out to me.

Release and demo time

I guess there is not a huge audience of M5StickS3 users who need a Hungarian fine-tuned model for it, but I did release the actual model:

https://huggingface.co/realhidden/whistle-finetuned-hu

My first Hugging Face release as well. :3 It is CC BY-NC 4.0, non-commercial only, because the WorldSpeech data it learned from is.

The two files you can flash onto the actual device to test it out (each is a complete image for address 0x0, so it replaces whatever is on the stick: esptool.py --chip esp32s3 write_flash 0x0 m5whistle-hu.bin, or straight from the browser with esptool-js):

• Hungarian one: m5whistle-hu.bin
• English one: m5whistle-en.bin

And a web demo, since it's easier to test it out.