stormregion
Keep flyin'
Playing around with the local AI stack for the last couple of months, TTS or Speech-to-Text was not my main focus. I installed Voicebox in the past, cloned my voice, made myself speak Korean, but it was slow and painful and not developer-friendly.
Then during my PTO I did see Qwen3-TTS vs dots.tts for local AI voice cloning by No place like localhost and the idea was born to create an end-to-end voice cloning PoC from source material. I did not want to use a 10-sec sample to generate a couple of lines, I wanted to see how mature this part of the LLM stack is.
After reading a bit, the plan is kind of simple: backpropagation on a pre-trained model, generating a LoRA (and/or baking it into the model itself).
The "core" part of the training is quite simple to be honest.
# Load the base Qwen3-TTS model in bf16
qwen3tts = Qwen3TTSModel.from_pretrained(
args.model,
device_map=device,
dtype=torch.bfloat16,
attn_implementation="sdpa",
)
model = qwen3tts.model
# Freeze the original model weights
for p in model.parameters():
p.requires_grad = False
# Add small trainable LoRA adapters
from peft import LoraConfig, get_peft_model
targets = ["q_proj", "k_proj", "v_proj", "o_proj"]
# Also adapt the MLP layers to better learn voice/timbre
targets += ["gate_proj", "up_proj", "down_proj"]
lora = LoraConfig(
r=32,
lora_alpha=64,
lora_dropout=0.05,
bias="none",
target_modules=targets,
task_type="CAUSAL_LM",
)
model.talker = get_peft_model(model.talker, lora)
model.talker.print_trainable_parameters()
# Train only the LoRA parameters
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = AdamW(
trainable,
lr=2e-5,
weight_decay=0.01,
)
# Example training step
main_loss, sub_loss = losses(batch)
loss = main_loss + 0.3 * sub_loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
But the training is one thing; there is a whole flow that I created to get to the final result. But the TL;DR is: we need input data for the training (WAV files and transcripts).
Getting the training data
The input was a 5.1 Blu-ray of the Firefly series and Serenity the movie (treated as the last episode of the series). Let's pretend that we have this on disk, MKV format, all subtitles ripped. I wanted to have just series-specific lines; characters can have different tempo, pitch, and timbre even if it's the same actor.
The grind was real in here. To get (per character) transcribed lines, I did:
- FFmpeg: 16 kHz + 48 kHz mono WAVs (5.1 center channel) - voice is usually muxed into the center
- Demux the subtitle stream (text) or PGS bitmap track (pgsrip). Also considered OpenSubtitles.com at some point
- Strip SDH annotations (Subtitles for the Deaf and Hard of Hearing) and other noise from the text, like [Music], [Applause], etc.
And at this point you will have a (badly) synced subtitle track which is somewhat clean at least, and an audio file per episode. No clue on speaker, no separation, doing a cut of the timestamps will leave you with a lot of half-finished sentences.
Fixing the training data
The desync is an issue we can deal with. Running Whisper will tell you exactly which word is where, so you can build out a shift database - since it's not just a constant per episode shift.
Listening to a couple of samples in here, I noticed a lot of background music and noise, so I decided to do some quick research with Claude on what I could do and Demucs was the recommended way. That can separate the voice from the music and other background noise, so you can get a clean voice track. I used the default model, which yielded great results - in most cases. Where the noise was too loud I simply removed it from the training dataset.
Now we have a whole bunch of cleanish lines, a reasonable amount to manually label, so I started doing that for some time, but in the background I started a new search for automatic labeling.
I learned about the VoxCeleb dataset, which is awesome. I tried using anything built on top of that, but single-pass embeddings of ~3s lines are noisy and same-speaker similarity lands around 0.25-0.5 (not the textbook 0.7+).
I needed to use a small but useful trick to get it working:
from speechbrain.inference.speaker import EncoderClassifier
from sklearn.cluster import AgglomerativeClustering
import numpy as np
import torch
# Load pretrained ECAPA speaker-embedding model
speaker_model = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
run_opts={"device": "cpu"},
)
def embed_segment(audio):
"""
Split one dialogue line into overlapping crops,
embed each crop, then average them.
"""
n = len(audio)
crops = [
audio[0:int(0.6 * n)],
audio[int(0.2 * n):int(0.8 * n)],
audio[int(0.4 * n):n],
]
embeddings = []
with torch.no_grad():
for crop in crops:
emb = speaker_model.encode_batch(
crop.unsqueeze(0)
)
embeddings.append(emb.squeeze().numpy())
embedding = np.mean(embeddings, axis=0)
# normalize for cosine distance
return embedding / np.linalg.norm(embedding)
# Create one speaker embedding for every dialogue segment
embeddings = [
embed_segment(segment_audio)
for segment_audio in clean_segments
]
X = np.stack(embeddings)
# Group similar voices together
clustering = AgglomerativeClustering(
n_clusters=None,
metric="cosine",
linkage="average",
distance_threshold=0.78,
)
cluster_ids = clustering.fit_predict(X)
# ...
# Small clusters are often fragments/noise.
# Merge them into the most similar large speaker cluster.
for small_cluster in small_clusters:
small_centroid = mean_embedding(small_cluster)
nearest_cluster = min(
large_clusters,
key=lambda cluster:
cosine_distance(
small_centroid,
mean_embedding(cluster),
),
)
if cosine_distance(
small_centroid,
mean_embedding(nearest_cluster),
) < 0.86:
merge(small_cluster, nearest_cluster)
I did some manual checking, seemed fine (but from the final results I still think there is a bit of mismatch somewhere).
Another source that I found is https://central-cortex.livejournal.com/, which had a stage-play-like version of the episodes that matched 95%+ with the auto-clustering.
Fun fact, none of the sources matched perfectly: Whisper, official subtitles, stage play, OpenSubtitles.
Then the boring part, the training
Once the pipeline was in order, I could start to do the plumbing and training. At this point I had 100 GB of audio, random transcripts, more WAVs than I ever wanted to deal with. Running it on my dual RTX 3060s was simple: the first run ran overnight, and once I had a working v1 I could throw Claude at it. The last run (I did around 12–14 rounds) was 1.5 hours long.
Just as a side note, these are the stats:
Subtitle cards, 14 episodes + film
9,452 100.0% 73,150w subtitle cards, 14 eps + film
8,353 88.4% 66,131w a speaker name could be aligned −1,099
6,799 71.9% 53,960w that speaker is one of the ten crew −1,554
3,046 32.2% 26,293w clean: CER ≤ 0.15, rel_db ≥ 18 −3,753
903 9.6% 10,307w recut into whole sentences −2,143
580 6.1% 7,459w survived clip_quality → trained on −323
The recut stage looks like the second-biggest cull but mostly isn't — it merges consecutive cards into whole sentences, so the card count drops 70% while words only drop 61%. The cleanliness gate is the real loss.
Per character, words spoken vs what reached training:
┌───────────────┬────────┬───────┬───────┬───────┬──────┬───────┐
│ │ series │ film │ lines │ clips │ min │ kept │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Mal Reynolds │ 17,574 │ 2,406 │ 2,360 │ 188 │ 14.3 │ 13.0% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Simon Tam │ 5,688 │ 316 │ 814 │ 56 │ 4.0 │ 10.8% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Jayne Cobb │ 4,829 │ 450 │ 642 │ 57 │ 4.9 │ 15.3% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Inara Serra │ 4,467 │ 227 │ 597 │ 54 │ 3.9 │ 14.8% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Wash │ 4,410 │ 236 │ 560 │ 47 │ 3.8 │ 14.4% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Zoe Washburne │ 3,459 │ 493 │ 483 │ 46 │ 3.2 │ 13.9% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Kaylee Frye │ 3,372 │ 309 │ 501 │ 52 │ 3.5 │ 14.3% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Shepherd Book │ 2,169 │ 117 │ 313 │ 36 │ 2.9 │ 19.7% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ Saffron │ 1,813 │ — │ 237 │ 18 │ 1.6 │ 14.5% │
├───────────────┼────────┼───────┼───────┼───────┼──────┼───────┤
│ River Tam │ 1,498 │ 127 │ 292 │ 26 │ 1.8 │ 15.3% │
└───────────────┴────────┴───────┴───────┴───────┴──────┴───────┘
One caveat you should know before quoting the film column. Speaker alignment covers 95.8% of series cards but only 42.4% of the film cards (560 of 1,321) — the Serenity fan transcript aligned badly. The film actually has 8,908 spoken words; only 4,720 got a name. So film counts are a floor and the series-vs-film ratio is unreliable.
Final result
For copyright reasons I cannot share the model or the training data, but the final results are here, you can be the judge of how good it is. If you want the easy-mode way of doing this, just use a clean voice clip and the clone mode (test that here: https://huggingface.co/spaces/Qwen/Qwen3-TTS).
My result, two lines per character:
Considering this is just a PoC, I do think it's better than I hoped for. Also I worked mostly with GLM 5.3 on the code side, which just came out, and Qwen 3 TTS on the training and LoRA side.