Since I love a challenge when it comes to cramming anything onto the small M5Stick devices I own, I took it on.

Model running on the stick

Size optimization is always like saving a hot air balloon from crashing: deciding what needs to be kept on the "ship" and what can be thrown out. And then just starting to throw things out. :D Cramming everything onto an M5StickS3 with an ESP32-S3 is always a challenge. 8 MB of flash and 8 MB of PSRAM are always tight.

The first challenge is that the C++ engine that runs Whistle on CPUs (libneedle) only ships as a compiled library. The JAX training code is open, the inference engine is not. Rebuilding things with AI from the outside seems to be this week's topic anyway, with someone recreating the Adobe Suite using AI. I won't publish the implementation, since keeping the engine closed was the authors' original intent. I will only publish compiled builds.

After the initial code was created, the partition table looked like this:

The packed EN model is 7,584,960 bytes, which needed some fiddling and throwing away a couple of useful things. Remember, the base model is 16.9 MB, but the stick has 8 MB of flash in total, and only about 7.3 MB of that is left for the model.

The engrams do help with decoding, but we don't have the space for them, and the decoder is actually trained to be cut down this way.

On my small test set (16 English sentences, 168 words, read by macOS text-to-speech), the official engine scored 8.9% WER with the full model, 6.5% at depth 2 (awesome, but on a set this small probably an outlier), and 9.5% at depth 2 without engrams. My reimplementation running the packed blob scored the same 9.5%, so the C port matches the official engine on the configuration that fits.

And running it on the stick turned out to be a mess — hitting the second common issue: CPU speed.

Speeding up the model

I had never used the S3's advanced "AI features", described in the official docs as:

AI Acceleration: Includes vector instructions and dedicated SIMD (Single Instruction, Multiple Data) capability, which significantly speeds up neural network computing and signal processing tasks

At this point, processing took 2.4× the audio duration, around 25 seconds to transcribe a ten-second input, without listening to anything else during that time. That is ... how to say it kindly ... not the best experience. I am not an expert in this field, but Opus 5.5 and GPT 6.1 Sol had some ideas, and I threw in my ideas as well. Teamwork in 2026, right? :D

I can explain my part, but since I like to understand AI code as well, I did go into the details of all of this. Learning the tricks always helps with future projects.

The changes were the following:

1. Silence gate: silent or steady-noise audio is rejected right after feature extraction (~0.5 s), before the encoder runs.

2. Doing less work

   • Output head: scoring all 6955 tokens with 4-bit weights at every step was expensive.
   • A 2-bit sketch, later 1-bit (signs only), shortlists 32 candidates, which get exact scores.
   • Search: beam 5 → 3 → greedy by default.
   • Normalizer: greedy decoding computes the log-sum-exp over the top 6 candidates instead of all logits.
   • Pause squeezing: pauses of 300 ms or more are shortened to 160 ms in the feature domain, giving about 20% fewer encoder frames on natural speech.
   • Small wins:
     • Text-token scans over a prefix.
     • Lookup-table sigmoid/exp.
     • Two real frames per FFT.
     • Angle-addition position embedding instead of sinf/cosf per element.

3. Memory: where data lives decides the speed

   • The hierarchy: internal SRAM is fast, but only ~30 KB is free. PSRAM bandwidth was ~56 MB/s. Flash reads come through the same cache.
   • Hot buffers in internal RAM, allocated first because that heap fragments. Later: avoiding slow RTC memory that the heap also labels "internal".
   • Decoder staging: decoder weights copied to PSRAM once per utterance (2.3× faster than flash), later with float norms precomputed.
   • Cache lines switched to 64 bytes at boot, at runtime. The prebuilt config uses 32, so PSRAM bandwidth went from 56 → 90 MB/s, reducing overall processing time by about 9%.
   • int16 encoder lanes (the 4-lane residual stream) halved that traffic, with no WER change.
   • Access patterns:
     • Cache-friendly loop orders.
     • Fused passes, so data is read once instead of 3–4 times.
     • Transposes done in tiles. The decoder memory's byte-per-line scatter went from 185 → 30 ms.

4. The SIMD unit (PIE)

   • int8 matmuls with Hadamard-rotated activations quantized in fixed point, so weights are never dequantized to float.
   • int8 attention (head-major K/V) and int16 Kronecker products for the HadamardMLP, with operands held in registers.
   • Bit-plane kernels: the 2-bit codebook [−127, −38, 38, 127] is exactly −127 + 89·b0 + 165·b1. A dot product therefore comes straight from shifting and masking the packed bits, with no table lookup and identical results. Same trick for the 1-bit head.
   • Row mode: one activation scale per frame, with group norms folded into the weights, so each output row is a single accumulator chain.
   • Measured on the chip, which shaped all of the above:
     • A 128-bit load costs 2 cycles.
     • Reading out the accumulator costs about 9 cycles.
     • So the kernels are designed around loads and readouts, not multiplies.

5. Two cores

   • wh_parallel everywhere: features, mHC passes, attention prep, activation quantization, decoder memory.
   • Split choice: splitting matmuls by output rows rather than frames, so the cores don't both decode the same weights.
   • Ceiling: two cores give only ~1.35× the speed, because they share the PSRAM bus.

6. Compiler and FPU lessons

   • O3 + IRAM for hot kernels. GCC 8 most likely miscompiled one fused kernel at O3 early on (garbage text until it was built at O2). After it was rewritten, O3 was fine.
   • Float compares and branches are slow: integer abs-max and the hardware rounding instruction (round.s) replaced them in the quantizers.
   • GCC 8 has no instruction scheduling for this core, and its multiply-add forces register copies. Hand-interleaved SiLU, norm and elementwise loops: SiLU went from 47 → 29 cycles per element.
   • esp-dsp's S3-optimized FFT for feature extraction.

And for anyone trying to reproduce this, these were the fails:

• DCache preload (no extra bandwidth).
• Branch-free sigmoid via float selects (slower).
• Dropping encoder layers (WER 49–100%).
• Hardware cache autoload (hangs the chip).
• DMA double-buffering.
• Turning off multiply-add contraction globally (slower).
• Interleaved row blocks between the cores.
• A smaller head sketch (costs WER).
• Streaming without look-ahead (WER 50–90%).
• An early-stop guard based on attention position.

As it turns out, the S3's advanced AI features are just a set of 128-bit vector (SIMD) instructions — up to 16 int8 multiply-accumulates per instruction, no NPU or matrix unit — which is sad, but still, they can help a lot. Interestingly enough, the easiest ideas (mine) yielded some of the largest wins, like not processing silence (duh), RAM and allocation magic I learned from past projects (ESP32-specific), and doing things in parallel (another you-don't-say).

With all of this, processing went from 2.4× the audio duration to about 0.85× — faster than real time, using both cores. (The two-core split was there from the very first version. It just never gave more than ~1.35×, because both cores share one PSRAM bus.)

My pet benchmark that I used for this (processing time on the stick, greedy decoding, the default):

I was a happy camper once we got these results. At this point, I just directed the LLMs and didn't really touch any code. I know, vibecoding, but the question was: can I squeeze the code onto the stick without butchering everything, while running in real time?

Let's teach Whistle some Hungarian

Hungarian is a hard language. Words are long and carry many suffixes, so a single wrong letter makes a whole word wrong, and WER punishes that hard. CER (character error rate) shows better how close a transcript really is, so it matters more than WER in some ways. Since the original code includes LoRA fine-tuning options for other languages, I started doing just that. I did not know which datasets to use, but there were some sources I could use (I bet there are others as well), plus the special sauce: my own lectures.

Nothing crazy here. After a couple of rounds, I did have a model. The training ran with the same 2-bit quantization and the same cut-down 2-layer decoder as the stick, so what is measured is what runs on the device (greedy decoding, 1000 test clips per set, WER / CER):

Since it took hours, I did cheat and take inspiration from Mimo's dashboard. Mine looks like this: Dashboard. An eye-watering €0.81 of GPU electricity was burned on the project (3.23 kWh at €0.25/kWh). :D

As I said, nothing special here, but now I do understand a bit better why everyone wants good-quality datasets. More epochs on the same data stopped helping after run E. Introducing each new dataset did yield improvements, so if someone has 4 million hours of hand-annotated Hungarian voice recordings, feel free to reach out to me.

Release and demo time

I guess there is not a huge audience of M5StickS3 users who need a Hungarian fine-tuned model for it, but I did release the actual model:

https://huggingface.co/realhidden/whistle-finetuned-hu

My first Hugging Face release as well. :3 It is CC BY-NC 4.0, non-commercial only, because the WorldSpeech data it learned from is.

The two files you can flash onto the actual device to test it out (each is a complete image for address 0x0, so it replaces whatever is on the stick: esptool.py --chip esp32s3 write_flash 0x0 m5whistle-hu.bin, or straight from the browser with esptool-js):

• Hungarian one: m5whistle-hu.bin
• English one: m5whistle-en.bin

And a web demo, since it's easier to test it out.

After searching for an alternative, I found eul, which looked awesome for my use case. The only thing was that its last release was years old, so it needed some love for today's Macs. GitHub is awesome in this sense: you can check out the forks through the network graph, so I could go fishing for the best version to use as a development base. :D

In this case, that meant multiple forks:

• https://github.com/miaoweiwei/eul
• https://github.com/chrsomle/eul
• https://github.com/StoneOlo/eul
• https://github.com/sclarkca/eul

Everyone had added something useful, and after the great merge of different branches, version 2.1.0 was born. Testing it for a week gave me the confidence that it would work well for my use case. Simple yet powerful stats, a modern-ish build of the old app, and some fixes here and there.

Act Two — Connecting multiple machines

Ad-hoc networks, connections, and clever networking tricks have always had a special place in my random projects. I even managed to sell Ábel on the idea of creating an ad-hoc tracing service in Node.js five years ago. Check out the project; it's still awesome: peek.

The idea is the same as it was back then: give a service a shared secret, and it automagically becomes part of a distributed network where different machines find each other automatically.

What's a shared secret? In this case, it's a pairing code you enter on each machine you want to connect. Machines with the same code can find each other and read each other's stats. No account or central registration needed. Anyone with that code can join the group, so treat it like a password.

Since I know more about networks and the technologies involved now, I split the job into two parts:

• Local discovery: same Wi-Fi / LAN, talk directly.
• Remote discovery: different networks, meet somewhere on the internet.

Both run at the same time, all the time.

One secret, three keys

Everything starts with the shared secret. A generated one looks like k7mqp-x3ndr-h9wza-2vtfe: 4 groups of 5 characters, from an alphabet without look-alikes (no 0/o, no 1/l/i), so you can type it on the other machine by hand without cursing. That's roughly 99 bits of randomness, assuming the characters are chosen uniformly — plenty for this use case.

The secret itself never goes on the wire. It gets stretched with PBKDF2 (200k rounds), then HKDF derives separate values, each with its own label. This is the way.

PBKDF2 and HKDF, without the alphabet soup PBKDF2 turns the secret into key material through repeated calculations. That makes each password guess more expensive for someone trying to crack it.

HKDF then derives separate values for separate jobs. Think of it as making three different keys from the same starting material: one for local discovery, one for the internet meeting room, and one for encryption. Seeing a discovery value doesn't give you the encryption key.

Actual copy-paste, starting with the master key material produced by PBKDF2:

let ikm = SymmetricKey(data: master)

func derive(_ label: String) -> SymmetricKey {
    HKDF<SHA256>.deriveKey(
        inputKeyMaterial: ikm,
        info: Data(label.utf8),
        outputByteCount: 32
    )
}

func hex(_ key: SymmetricKey) -> String {
    key.withUnsafeBytes {
        $0.prefix(16).map { String(format: "%02x", $0) }.joined()
    }
}

return Keys(
    lanToken: hex(derive("lan")),
    topic: "eul-peer/\(hex(derive("topic")))",
    seal: derive("seal")
)

I do not code Swift, but Opus does. :D

• lan — a token to recognize my peers on the local network.
• topic — the meeting room name on the internet.
• seal — the ChaChaPoly key that encrypts every snapshot.

A snapshot is simply one sample of a machine's current stats, packaged into a message.

Why the PBKDF2 part? Because the topic isn't private. The broker can see eul-peer/3f9a..., and someone who learns that value can test guesses to see which secret produces it.

With a strong generated secret, that is hopeless anyway. But people do type password123 into things, so the slow stretching is the seatbelt (and the UI warns below 12 characters). It makes guessing slower; it doesn't turn a terrible password into a good one.

Local discovery: Bonjour

Apple gave us Bonjour years ago, and the Network framework makes using it a couple of lines. Yet understanding what's going on underneath can take an afternoon. It's a pretty clever way to advertise services on the local network, and something I think everyone should know a little about.

What's Bonjour? Bonjour is Apple's implementation of zero-configuration networking. Its local discovery uses multicast DNS (mDNS) and DNS Service Discovery (DNS-SD).

Instead of asking a central server where a service lives, machines send discovery messages to a shared address on the local network. Nearby machines can announce what they offer and answer requests. That's how an app can find another app without you typing an IP address.

A service announcement includes a service type, connection details, and optional metadata in a TXT record. Here, that metadata contains the pairing token. Discovery generally stays within the local network, and the network has to allow multicast traffic.

Every eul instance does two things:

• Advertises an _eul-peer._udp service, with the lan token in the TXT record.
• Browses for the same service type and keeps only the results where the token matches.

var txt = NWTXTRecord()
txt[Self.tokenKey] = token
listener.service = NWListener.Service(
    type: Self.serviceType,
    txtRecord: txt
)

When a match shows up, it opens a UDP flow to that peer, and every 5 seconds a sealed snapshot goes straight to it. No broker, no internet.

Why UDP? UDP sends individual packets without setting up the reliable, ordered stream you get with TCP. A packet can get lost or arrive late, but for stats refreshed every 5 seconds, the next update is usually more useful than retrying an old one.

The token is a derived value, not the secret. Broadcasting it doesn't directly reveal the key needed to read the stats. It does let someone recognize machines in the same group and test password guesses, which is another reason to use the generated secret.

Two small gotchas I ran into:

• Results come back once per interface (Wi-Fi + Ethernet + whatever VPN you have), so the first endpoint per service name wins. Otherwise, you send everything 3 times.
• You will find yourself. Bonjour can also rename your service after a name conflict, so the "is this me?" check uses the name the system actually registered, not the host name.

Remote discovery: someone else's MQTT broker

Different networks are where it gets interesting. NAT, firewalls, no local multicast, and it's always a pain in the ass to provide a public service on the internet for an open-source project.

Why can't the machines just connect directly? Your home router usually uses NAT: multiple devices share one public IP address, while each device has its own private address inside the network. Outgoing connections are straightforward, but an incoming connection generally needs port forwarding or some other way through.

A relay avoids that problem because both machines connect outward to a server they can reach.

Then it clicked: public MQTT brokers. Free, open, and built exactly for "many clients publish and subscribe to a topic."

What's MQTT? MQTT is a lightweight messaging protocol. Clients connect to a server called a broker, which passes messages between them.

A client publishes a message to a named topic, and clients subscribed to that topic receive it. Think of the topic as a meeting room: everyone joins the same room, and the broker forwards whatever gets posted there.

The broker doesn't need to understand the message contents. That makes it useful as a relay for encrypted data.

The topic is the derived eul-peer/<hex>. Everyone with the same secret subscribes to the same room and publishes a sealed snapshot into it every 5 seconds.

The broker sees encrypted bytes on a random-looking topic. Without the key, it can't read the stats or create a new message that passes the authentication check. It can still see connection details and message timing, and it can drop, delay, or replay messages.

What's ChaChaPoly? CryptoKit's ChaChaPoly combines ChaCha20 encryption with Poly1305 authentication. Encryption hides the contents; authentication checks that the message was created by someone with the key and hasn't been altered.

A modified message, or one created without the key, fails that check and gets dropped. An unchanged copy of an old valid message still passes the cryptographic check, so the app also needs duplicate and freshness checks.

Since all peers share the key, this proves membership in the group, not the identity of a particular machine.

The first version used one broker with a fallback list, which was suboptimal. :P Machine A falls back to broker #2, machine B is happy on broker #1, and they will never meet.

So now every instance joins all of them at once:

private static let brokers = [
    MQTTClient.Broker(host: "broker.emqx.io", port: 8883),
    MQTTClient.Broker(host: "broker.hivemq.com", port: 8883),
    MQTTClient.Broker(host: "test.mosquitto.org", port: 8886),
]

The MQTT client itself is a minimal, hand-written MQTT 3.1.1 implementation: QoS 0 only, one topic. Pulling in a full MQTT library for publish + subscribe felt like overkill.

What's QoS 0? MQTT has different delivery levels, called Quality of Service (QoS). QoS 0 sends messages without MQTT-level acknowledgements or retries, so delivery isn't guaranteed.

That's a reasonable tradeoff for periodically refreshed stats: if one snapshot goes missing, another should arrive a few seconds later.

These are public shared brokers, so I'm relying on someone else's infrastructure. Handy for this project, but not something I'd treat as a guaranteed service.

Same snapshot, many times

With Bonjour plus three brokers, the same snapshot can arrive up to 4 times. Plus, the brokers can echo your own publications back to you.

Each snapshot carries a per-launch instance ID and the sender's timestamp, so the receiver:

• Drops its own messages (same instance ID).
• Drops anything older than 60 seconds, limiting how long an old snapshot can be replayed.
• Keeps only the newest sentAt per peer. Duplicates from the slower paths just update "last seen" and the "via LAN / via relay" badge.

The instance ID is a temporary identifier generated when the app starts. It lets the receiver distinguish running peers without needing a central registry.

The timestamp check also assumes the machines' clocks are reasonably in sync. It isn't complete replay protection: a recent valid message can still be repeated inside that window, although it won't replace a newer snapshot.

After roughly three missed updates — a 16-second timeout — the peer disappears from the list. A restarted peer simply shows up as a new one, and the old entry expires on its own. No goodbye messages, no state to clean up.

I do love this solution, so I think I'll use it next time as well. Elegant, small(ish), and simple enough to have just a couple of moving parts.

The Linux side

Once I had the Mac solution tested, I wanted to create a Linux CLI port. Plain C99, runs on a Pi, and speaks the same protocol, so it shows up in the Mac panel like any other peer.

It only does the broker part, over plain TCP on port 1883, using the same brokers and topic. Adding TLS to a dependency-free C binary was not a yak I wanted to shave. The Bonjour portion would also need mDNS/DNS-SD support, so I left that out as well.

TCP, TLS, and the encrypted payload TCP provides a reliable, ordered connection between the client and broker. TLS adds encryption to that connection and, with proper certificate validation, verifies the broker's identity.

The snapshots are already encrypted end-to-end: the sending machine encrypts them, and the receiving peers decrypt them. So even over plain TCP, the stats themselves remain encrypted.

TLS still adds something useful, though. Without it, someone watching the connection can see the MQTT topic and protocol traffic, and an active attacker can interfere with the connection. Payload encryption protects the stats; TLS protects the connection to the broker.

So skipping TLS is a tradeoff in the Linux port, even though the payload is already encrypted.

The crypto had to be byte-compatible with CryptoKit, so I asked Opus to write PBKDF2, HKDF, and ChaChaPoly in C and test them against the RFC vectors. On macOS, there's also a cross-check that seals a message on one side and opens it on the other.

What's an RFC test vector? It's a published example with known inputs and expected outputs. Feeding those inputs into your implementation should produce exactly the documented result.

The cross-check then tests whether the C implementation and CryptoKit can actually exchange encrypted messages. Useful checks, though passing them isn't the same as a full security review.

Open-source project

This started as an open-source project, and I'm keeping it that way, so you can see my exact modifications:

And just to clarify: a FREE and OPEN-SOURCE iStat Menus replacement that needs no server of your own, no accounts, and no monthly bill to check on your other machines remotely.

Local peers talk directly. Remote peers use public MQTT brokers as relays. Same secret, same group, stats in your menu bar.

Feel free to contribute, fork, or just use it. I hope it helps you as much as it helps me.

I use Minimax H3 for most of the mass-scale image generation, but for one-off images ChatGPT and Sora are way better and faster. They follow specific style hints and keep context. The only problem with that workflow is that it's either API cost (on top of the generic monthly plan) or a manual process. I know it doesn't make sense for them to provide it as an MCP, but that should not stop us from rigging something together.

Chrome extension to interact with a chat

The idea is not new: grab a ChatGPT tab, wire in some automation code, send messages and read the results back. I did something really similar in 2023 during my class, but this time I went the proper way: a Chrome extension.

The extension itself is the "connector". Using simple HTML selectors, it can:

• write messages into the chat
• read messages back
• download images

To let the MCP work with the browser, you need to select a specific tab / chat (bottom right of the image). Once that chat / tab is locked, the MCP uses it for all queries. For a context switch, the extension can open a new chat, and by sending a sample query it can check whether the flow is in the receiving state. I used the string "Are you ready?", which, as a side effect, also tags most of those chats as "Ready check" or "Readiness check".

The MCP

Creating the MCP portion is the boring part. @modelcontextprotocol/sdk makes it super simple: throw in the tool definitions, package them into one Node.js project, register it in Claude, for example, and you are done.

Just for fun, here is what a tool definition looks like (actual copy-paste, some bootstrapping is not shown):

server.registerTool(
  "chatgpt_last_message",
  {
    title: "Read ChatGPT's last reply",
    description:
      "Return the newest reply in the marked ChatGPT conversation as markdown, without sending " +
      "anything. Waits if ChatGPT is still answering. Use after chatgpt_ask timed out, or to " +
      "read an answer produced by hand in the tab.",
    inputSchema: {
      timeout_seconds: z.number().int().min(10).max(3600).optional().describe("Default 600."),
      output_dir: z.string().optional(),
    },
  },
  async ({ timeout_seconds = 600, output_dir }) => {
    let res;
    try {
      res = await bridge.call("last", { expectImages: false }, { timeoutMs: timeout_seconds * 1000 });
    } catch (e) {
      return fail(`could not read the last reply: ${e.message}`);
    }
    return textReply(res, output_dir);
  }
);

And the Claude Code wiring:

{
  "mcpServers": {
    "chatgpt-image": {
      "command": "node",
      "args": ["/absolute/path/to/chatgptimagemcp/server/src/index.js"]
    }
  }
}

Routing challenge

The only interesting part, which gave some good old patterns a new shine, was selecting a leader in a distributed system and orchestrating multiple clients. Man, these bring back memories of the good old university days.

Why the heck do we need this in the first place? The way the system is built, the Chrome extension connects to a local port, and not the other way around, to keep the extension simple. This simplification brings another problem: on the MCP side, only one process can listen on a specific port. That works great, as long as you don't want to use it from multiple sessions. :D

When multiple sessions exist, the first MCP gets the port and can communicate with the extension, while everyone else fails at the socket listen. That failure is the trigger for another mode inside the MCP: the second instance connects to the first and asks it to proxy requests over to the extension.

If the "first" MCP instance dies for some reason, which can be as simple as ending the session, a leader election starts. It does the same "dance" as the first round: try to listen on the port, and fall back to being a peer if that fails.

Of course, the image was generated using the same flow. :D

Conclusion and remarks for others

Sometimes the more boring projects are the most useful. Since I built this MCP, I have used it almost every time I need "just a small graphic" or a rendered UML chart. The project is relatively easy to reimplement, so sharing the source does not make sense (and there is a bit of a "can I even do that considering the TOS?" question). Things not mentioned above are refusals and rate limits, and the fact that after 20-30 images in one chat the solution becomes less reliable, even if you scroll, etc. Maybe it's running the tab in the background, or some timing issue; I'm not sure. My solution is to open a new chat and keep grinding. :D

Demo

Then during my PTO I did see Qwen3-TTS vs dots.tts for local AI voice cloning by No place like localhost and the idea was born to create an end-to-end voice cloning PoC from source material. I did not want to use a 10-sec sample to generate a couple of lines, I wanted to see how mature this part of the LLM stack is.

After reading a bit, the plan is kind of simple: backpropagation on a pre-trained model, generating a LoRA (and/or baking it into the model itself).

The "core" part of the training is quite simple to be honest.

# Load the base Qwen3-TTS model in bf16
qwen3tts = Qwen3TTSModel.from_pretrained(
    args.model,
    device_map=device,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

model = qwen3tts.model

# Freeze the original model weights
for p in model.parameters():
    p.requires_grad = False

# Add small trainable LoRA adapters
from peft import LoraConfig, get_peft_model

targets = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Also adapt the MLP layers to better learn voice/timbre
targets += ["gate_proj", "up_proj", "down_proj"]

lora = LoraConfig(
    r=32,
    lora_alpha=64,
    lora_dropout=0.05,
    bias="none",
    target_modules=targets,
    task_type="CAUSAL_LM",
)

model.talker = get_peft_model(model.talker, lora)
model.talker.print_trainable_parameters()

# Train only the LoRA parameters
trainable = [p for p in model.parameters() if p.requires_grad]

optimizer = AdamW(
    trainable,
    lr=2e-5,
    weight_decay=0.01,
)

# Example training step
main_loss, sub_loss = losses(batch)
loss = main_loss + 0.3 * sub_loss

loss.backward()
optimizer.step()
optimizer.zero_grad()

But the training is one thing; there is a whole flow that I created to get to the final result. But the TL;DR is: we need input data for the training (WAV files and transcripts).

Getting the training data

The input was a 5.1 Blu-ray of the Firefly series and Serenity the movie (treated as the last episode of the series). Let's pretend that we have this on disk, MKV format, all subtitles ripped. I wanted to have just series-specific lines; characters can have different tempo, pitch, and timbre even if it's the same actor.

The grind was real in here. To get (per character) transcribed lines, I did:

1. FFmpeg: 16 kHz + 48 kHz mono WAVs (5.1 center channel) - voice is usually muxed into the center
2. Demux the subtitle stream (text) or PGS bitmap track (pgsrip). Also considered OpenSubtitles.com at some point
3. Strip SDH annotations (Subtitles for the Deaf and Hard of Hearing) and other noise from the text, like [Music], [Applause], etc.

And at this point you will have a (badly) synced subtitle track which is somewhat clean at least, and an audio file per episode. No clue on speaker, no separation, doing a cut of the timestamps will leave you with a lot of half-finished sentences.

Fixing the training data

The desync is an issue we can deal with. Running Whisper will tell you exactly which word is where, so you can build out a shift database - since it's not just a constant per episode shift.

Listening to a couple of samples in here, I noticed a lot of background music and noise, so I decided to do some quick research with Claude on what I could do and Demucs was the recommended way. That can separate the voice from the music and other background noise, so you can get a clean voice track. I used the default model, which yielded great results - in most cases. Where the noise was too loud I simply removed it from the training dataset.

Now we have a whole bunch of cleanish lines, a reasonable amount to manually label, so I started doing that for some time, but in the background I started a new search for automatic labeling.

I learned about the VoxCeleb dataset, which is awesome. I tried using anything built on top of that, but single-pass embeddings of ~3s lines are noisy and same-speaker similarity lands around 0.25-0.5 (not the textbook 0.7+).

I needed to use a small but useful trick to get it working:

from speechbrain.inference.speaker import EncoderClassifier
from sklearn.cluster import AgglomerativeClustering
import numpy as np
import torch

# Load pretrained ECAPA speaker-embedding model
speaker_model = EncoderClassifier.from_hparams(
    source="speechbrain/spkrec-ecapa-voxceleb",
    run_opts={"device": "cpu"},
)


def embed_segment(audio):
    """
    Split one dialogue line into overlapping crops,
    embed each crop, then average them.
    """
    n = len(audio)

    crops = [
        audio[0:int(0.6 * n)],
        audio[int(0.2 * n):int(0.8 * n)],
        audio[int(0.4 * n):n],
    ]

    embeddings = []

    with torch.no_grad():
        for crop in crops:
            emb = speaker_model.encode_batch(
                crop.unsqueeze(0)
            )
            embeddings.append(emb.squeeze().numpy())

    embedding = np.mean(embeddings, axis=0)

    # normalize for cosine distance
    return embedding / np.linalg.norm(embedding)


# Create one speaker embedding for every dialogue segment
embeddings = [
    embed_segment(segment_audio)
    for segment_audio in clean_segments
]

X = np.stack(embeddings)


# Group similar voices together
clustering = AgglomerativeClustering(
    n_clusters=None,
    metric="cosine",
    linkage="average",
    distance_threshold=0.78,
)

cluster_ids = clustering.fit_predict(X)

# ...

# Small clusters are often fragments/noise.
# Merge them into the most similar large speaker cluster.

for small_cluster in small_clusters:

    small_centroid = mean_embedding(small_cluster)

    nearest_cluster = min(
        large_clusters,
        key=lambda cluster:
            cosine_distance(
                small_centroid,
                mean_embedding(cluster),
            ),
    )

    if cosine_distance(
        small_centroid,
        mean_embedding(nearest_cluster),
    ) < 0.86:
        merge(small_cluster, nearest_cluster)

I did some manual checking, seemed fine (but from the final results I still think there is a bit of mismatch somewhere).

Another source that I found is https://central-cortex.livejournal.com/, which had a stage-play-like version of the episodes that matched 95%+ with the auto-clustering.

Fun fact, none of the sources matched perfectly: Whisper, official subtitles, stage play, OpenSubtitles.

Then the boring part, the training

Once the pipeline was in order, I could start to do the plumbing and training. At this point I had 100 GB of audio, random transcripts, more WAVs than I ever wanted to deal with. Running it on my dual RTX 3060s was simple: the first run ran overnight, and once I had a working v1 I could throw Claude at it. The last run (I did around 12–14 rounds) was 1.5 hours long.

Just as a side note, these are the stats:

Subtitle cards, 14 episodes + film

  9,452  100.0%   73,150w   subtitle cards, 14 eps + film
  8,353   88.4%   66,131w   a speaker name could be aligned      −1,099
  6,799   71.9%   53,960w   that speaker is one of the ten crew  −1,554
  3,046   32.2%   26,293w   clean: CER ≤ 0.15, rel_db ≥ 18       −3,753
    903    9.6%   10,307w   recut into whole sentences           −2,143
    580    6.1%    7,459w   survived clip_quality → trained on     −323

The recut stage looks like the second-biggest cull but mostly isn't — it merges consecutive cards into whole sentences, so the card count drops 70% while words only drop 61%. The cleanliness gate is the real loss.

Per character, words spoken vs what reached training:

  ┌───────────────┬────────┬───────┬───────┬───────┬──────┬───────┐
  │               │ series │ film  │ lines │ clips │ min  │ kept  │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Mal Reynolds  │ 17,574 │ 2,406 │ 2,360 │   188 │ 14.3 │ 13.0% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Simon Tam     │  5,688 │   316 │   814 │    56 │  4.0 │ 10.8% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Jayne Cobb    │  4,829 │   450 │   642 │    57 │  4.9 │ 15.3% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Inara Serra   │  4,467 │   227 │   597 │    54 │  3.9 │ 14.8% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Wash          │  4,410 │   236 │   560 │    47 │  3.8 │ 14.4% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Zoe Washburne │  3,459 │   493 │   483 │    46 │  3.2 │ 13.9% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Kaylee Frye   │  3,372 │   309 │   501 │    52 │  3.5 │ 14.3% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Shepherd Book │  2,169 │   117 │   313 │    36 │  2.9 │ 19.7% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ Saffron       │  1,813 │     — │   237 │    18 │  1.6 │ 14.5% │
  ├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
  │ River Tam     │  1,498 │   127 │   292 │    26 │  1.8 │ 15.3% │
  └───────────────┴────────┴───────┴───────┴───────┴──────┴───────┘

One caveat you should know before quoting the film column. Speaker alignment covers 95.8% of series cards but only 42.4% of the film cards (560 of 1,321) — the Serenity fan transcript aligned badly. The film actually has 8,908 spoken words; only 4,720 got a name. So film counts are a floor and the series-vs-film ratio is unreliable.

Final result

For copyright reasons I cannot share the model or the training data, but the final results are here, you can be the judge of how good it is. If you want the easy-mode way of doing this, just use a clean voice clip and the clone mode (test that here: https://huggingface.co/spaces/Qwen/Qwen3-TTS).

My result, two lines per character:

Download the MP3

Considering this is just a PoC, I do think it's better than I hoped for. Also I worked mostly with GLM 5.3 on the code side, which just came out, and Qwen 3 TTS on the training and LoRA side.

And yes, sometimes it's going to be a total AI slopfest, with the same UI elements, same texts, random missing features, etc. etc., but that can happen with humans as well — I do still remember Stack Overflow copy-paste reviews.

Anyway, my better half loves South Korean culture and bands, especially Stray Kids. Grabbing someone's sweet-spot topic for a vibe-coded project gets you almost halfway there. The other half is the idea, and if you pick the right representation, everything just snaps together.

I like doing 8bit / pixel art style GFX, since it hides most of the problems and gives 30-ish folks fond childhood memories, so there are some extra points in there. Since this is really a presentation of what technology can do, let me share my "master prompt" that yielded the whole flow, and how I worked from there.

Master prompt

Let's create a small tamagotchi game, where you can / should manage one of the Stray Kids singers. Include everyone in the game and make it entertaining.

Be creative, but truthful to the original game and characters. 8bit-like color graphics, and it should run on an M5StickC PLUS2 ESP32.

I want to have some music in the game and diverse minigames of some sort.

First step: create a DETAILED idea for the game. The device should be held upwards, which will make 240 x 135 for the playfield.

It has everything I just mentioned: the fun topic, the "anti-slopify filter" of 8bit, the nostalgic Tamagotchi factor and, of course, the "no mistakes" part with "make it entertaining". :D

This went into ChatGPT, and 23k characters later we got a decent Tamagotchi game skeleton: goal.md

Once this was ready, I just threw it into Z.ai GLM 5.1 with a "build it man" prompt and let that thread panic, for pleasure.

The other part of the grind was the graphics. A couple of prompts later, in the same ChatGPT session, I had a list of assets that I needed to grab for the game. At that time I did not have my image generator or ChatGPT image MCP, so copy-pasting prompts that ChatGPT generated into a new session, then grabbing the images from that session, took around 1–1.5 hours.

Since this was a late-night vibecoding session, I was watching something on the TV the whole time and just let this run in the background.

The results were great, but the image generator still cannot keep a nice grid for 8bit games, so that needed some extra code on GLM's side to crop the images.

Hyunjin

Once the characters were decent and every asset was generated, I connected the device itself and let it cook.

Once there was enough to see that we were heading in the right direction, I wanted some fine-tuning, a manual and so on, so v2 was born: v2.md. This was generated by Opus, since GLM is not that strong at planning, but great at execution.

Let me remind you that at this point, there had only been some light testing and the project had been started roughly 4 hours earlier. The timeline looks something like this based on git commits, once I started asking for those.

Background images and other assets turned out pretty decent as well:

Timeline of the project

26b8ca0 - Mon Aug 10 00:40:43 2026 +0200 : Illustrated manual (HTML + PDF) and v2 notes
b72b078 - Mon Aug 10 00:27:54 2026 +0200 : Browser build via Emscripten
685de90 - Mon Aug 10 00:17:51 2026 +0200 : QUICK MODE and an explicit reset
618d344 - Mon Aug 10 00:10:56 2026 +0200 : Fix a freshly reset pet being charged for every hour since the epoch
3253f40 - Mon Aug 10 00:08:02 2026 +0200 : Iteration 8: birthdays, collection screen, adaptive soundtrack
6ced6c8 - Mon Aug 10 00:03:57 2026 +0200 : Iteration 7: comeback week and the concert finale
b22ff29 - Sun Aug 9 23:54:16 2026 +0200 : Iteration 6: the remaining seven minigames
bc6ae7e - Sun Aug 9 23:43:25 2026 +0200 : Iteration 5: minigame framework and the three input-defining games
bad1ee2 - Sun Aug 9 23:33:57 2026 +0200 : Iteration 4: random event and dialogue engine
fe1cde9 - Sun Aug 9 23:23:36 2026 +0200 : Fix save being rewritten immediately after a wipe
8879a37 - Sun Aug 9 23:19:53 2026 +0200 : Iteration 3 (WIP): locations, daily schedules, activity scene
74f3594 - Sun Aug 9 23:07:57 2026 +0200 : Iteration 2: fix sprite transparency, all eight members, new locations
fbfa599 - Sun Aug 9 22:42:47 2026 +0200 : Pin upload/monitor port glob to /dev/cu.usb* so auto-detect skips Bluetooth
f0fbeee - Sun Aug 9 22:42:05 2026 +0200 : Iteration 1: playable dorm loop for M5StickC PLUS2

The next day

Testing, giving feedback and iterating on a project is the long grind. Almost everything so far followed the "simple" route of letting the LLM decide what to do, what is fun, and in some cases where to cut corners.

I tend to pick the simplest answer when something is vibecoded, and iterate on it later.

The result is OK, runs well on the device, and is somewhat fun-ish to play. But most importantly, it helped show how "easy" getting from an idea to a POC is nowadays.

The manual can be accessed here: PDF and HTML

And you can play the game here, but since it's a Tamagotchi game, the browser version is more like a POC than the real thing: game

There were some small(ish) features implemented since v2, including support for two extra languages — Korean and Hungarian — and wrapping it into an Expo app with shims, but nothing fancy.

TLDR

You can create fun projects from basically a one-liner, co-creating something that can be thrown away but can still create long-term inspiration for the next project.

And if it motivates someone to look into this crazy world as well, that's a plus.

Fan project / legal notice

SKZ-GOTCHI is an unofficial, non-commercial fan project made for fun, experimentation and as a technical demo. It is not affiliated with, endorsed by, sponsored by, or connected to Stray Kids, JYP Entertainment, or any related rights holders.

Stray Kids, member names, likenesses, trademarks, music, artwork and other associated intellectual property belong to their respective owners. This project does not claim ownership of any third-party intellectual property.

Now, that is interesting and all, and there were great products at the end of the day, but since I am not allowed to join any team (part of the ruleset), I needed to come up with my own idea. I always had this "two-factor authentication using sound and beeps" idea, where nearby machines can just beep to each other, creating ad-hoc channels. In 2024, some guys even ran with my idea for a TDK at BME. But I never tried it myself, and since it's a small enough problem to solve in a day, let's make it a bit more challenging: having JUST the device and somehow making a normal modern computer communicate with it, without using any other channel. A good metaphor is probably a message in a bottle, where you want to read the message but first need to somehow get it out of the bottle.

The M5Stick I used for this one is the M5StickC PLUS2, so there are WAY more limitations than you want to admit. :D

Communication idea

Breaking down this challenge is the key to solving it. So:

1. Somehow bootstrap the channel between the computer and the stick (in a secure-ish way)
2. Write the message (using sound)
3. Keep it on the device.

And when the time for the readback comes, same-ish process:

1. Bootstrap channel
2. Read message (using sound)

The bootstrap part is probably more interesting than the sound part, since that one is more like an old dial-up modem in a simplified way.

Bootstrapping

Let's assume there is no internet and we don't want to use Bluetooth or Wi-Fi, to make it more challenging. That leaves a couple of ways a computer can receive input:

1. Physical wire things, like USB or so
2. Camera, let's assume we are talking about a laptop
3. Microphone

Since the camera seems like a good idea, I started with the obvious "does not need anything else, just the OS" approach with QR-code-based communication. Well... a 1.14-inch, 135 × 240 LCD is not the largest thing in the world. :D I managed to squeeze 515 bytes of code into a 97×97 pixel grid, and mostly failed to read it as a QR code.

This is one of the test QR codes:

It can barely fit the 515-byte payload, but to be honest, it's mostly non-working, and the payload itself got larger.

That was the stage where I decided to do a multi-stage deal. The first stage is just minimal, small code that can receive data: small footprint, slow data transfer, but enough to receive the second stage, which is the actual code that we need for the fast(er) communication. Man, if that LCD had been just 50% larger, it might have worked.

Since there is no "audio to code" type tool (as far as I know) on a standard machine, and I did not want to mess around with implementing a USB protocol AND writing some things to my Mac, I fell back to let's type in the first stage.

Still, I needed a first stage. My solution was to create a super-minimal HTML page for this:

data:text/html,<i onclick=\"a=new AudioContext;N=a.sampleRate/100|0;navigator.mediaDevices.getUserMedia({audio:1}).then(s=>{sp=a.createScriptProcessor(256);a.createMediaStreamSource(s).connect(sp);st=b=n=k=x=0;o='';sp.onaudioprocess=e=>{for(v of e.inputBuffer.getChannelData(0)){x+=v*v;w=st?N:44;if(++k<w)continue;st&&x>M&&(M=x);q=x>(st?M/2:w/1e3);k=x=0;if(!st){st=q;M=0;continue}b=b*2+q;if(++n&7)continue;st<2?st++:st<3?b!=85?st=0:st++:st<4?(l=b,st=4):st<5?(l=l*256+b,st=5):st<6?(o+=String.fromCharCode(b),--l?0:st=6):0;b=n=0;if(st>5)this.innerHTML=o}};sp.connect(a.destination)})\">RECV</i>

Which looks awesome on the device BTW (a bit older screenshot, but still):

I know, I know. It will need some explaining:

const audioContext = new AudioContext();

const preambleWindowSamples =
    Math.floor(audioContext.sampleRate / 100);

navigator.mediaDevices
    .getUserMedia({ audio: true })
    .then(stream => {
        const processor =
            audioContext.createScriptProcessor(256);

        const source =
            audioContext.createMediaStreamSource(stream);

        source.connect(processor);

        let state = 0;
        let currentByte = 0;
        let bitCount = 0;

        let sampleCount = 0;
        let energy = 0;

        let output = '';
        let maxPreambleEnergy = 0;
        let remainingLength = 0;

        processor.onaudioprocess = event => {
            const samples =
                event.inputBuffer.getChannelData(0);

            for (const sample of samples) {
                energy += sample * sample;

                const windowSize = state
                    ? preambleWindowSamples
                    : 44;

                sampleCount++;

                if (sampleCount < windowSize) {
                    continue;
                }

                if (
                    state &&
                    energy > maxPreambleEnergy
                ) {
                    maxPreambleEnergy = energy;
                }

                const threshold = state
                    ? maxPreambleEnergy / 2
                    : windowSize / 1000;

                const bit = energy > threshold;

                sampleCount = 0;
                energy = 0;

                /*
                 * Wait for the first energetic block.
                 */
                if (!state) {
                    state = bit;
                    maxPreambleEnergy = 0;
                    continue;
                }

                /*
                 * Assemble one byte, MSB-first.
                 */
                currentByte =
                    currentByte * 2 + bit;

                bitCount++;

                if (bitCount & 7) {
                    continue;
                }

                /*
                 * Parse protocol bytes.
                 */
                switch (state) {
                    case 1:
                        // Guard byte.
                        state = 2;
                        break;

                    case 2:
                        // Sync byte = 0x55.
                        if (currentByte !== 0x55) {
                            state = 0;
                        } else {
                            state = 3;
                        }
                        break;

                    case 3:
                        // Length high byte.
                        remainingLength =
                            currentByte;

                        state = 4;
                        break;

                    case 4:
                        // Length low byte.
                        remainingLength =
                            (remainingLength << 8) |
                            currentByte;

                        state = 5;
                        break;

                    case 5:
                        // Payload.
                        output +=
                            String.fromCharCode(
                                currentByte
                            );

                        remainingLength--;

                        if (remainingLength === 0) {
                            state = 6;
                        }
                        break;
                }

                currentByte = 0;
                bitCount = 0;

                if (state === 6) {
                    this.innerHTML = output;
                }
            }
        };

        processor.connect(
            audioContext.destination
        );
    });

A small state machine with a simple "on-off" signal that is converted into one tone or silence. One long flow, no error correction, no checks, nothing. But at least it can wait for the first part of the transfer, the long beep, and then the sync. Also, it's super slow, since it should be somewhat robust to other noises, hence the 256 samples turning into 1 byte.

Giving a small ChatGPT explanation of this:

More than 3 minutes just to put 2413 bytes of HTML from point A to point B. The beeper on the device is not the best, and during the Hackathon I annoyed everyone enough that, for such a short time, this should be enough. I bet some improvements can be made.

The resulting HTML is a bit uglier than the original (CSS and debug strings cut from that), but this is how the page should have looked:

Stage 2

The Stage 2 interface is basically a huge textarea with 2 buttons (functions). The first is "COPY": whatever the user enters into the textarea gets converted to sound and played. For example, this 25,187-byte-long fan fiction in the Terminator universe. Yes, 1.4 hours of beeping for roughly 5 bytes/sec. It uses the fancier, more robust, 4-tone version: checksum, everything fancy that I did learn at the university. :D ChatGPT explanation:

The beeper itself is limited, the microphone is limited, and it's an open-air communication protocol hacked together in 2413 bytes. Once the HTML is loaded onto a machine, it can act both as a receiver and sender, so if you are lucky, you only need to load the initial 3 minutes once.

A function of the stick itself is to store data persistently, so you can record messages (stored as messages, not voice, since we just have a couple of usable kBytes).

A hint of style

Of course, presentation is half of the story in Hackathons, so slapping a Terminator story around it (keeping the communication secure-ish by using an obscure sound protocol, no internet, etc.), adding some design, and we got a crazy Hackathon project. :)

Credits

• OpenCode (Big Pickle, DeepSeek V4 Flash)
• Z.AI GLM 5.2
• Qwen Image 2.0
• Me as the host, the guy orchestrating agents and pushing random buttons on a small beeping device