I wanted to know whether a Japanese voice clone could run inside a static website. No API, no GPU server, no bill for each sentence. Just HTML, JavaScript, a WASM runtime, and model files.
The short clip above is pre-generated from the 5.3MB model; the demo below runs it live. The rest is how I got there, what broke, and where the boundary sits.
6.5 minutes of my voice
I recorded 122 Japanese utterances, 393 seconds in total, about 6.5 minutes. The recordings were 48kHz mono. The training pipeline resampled them to 22.05kHz, the rate the model outputs.
Six and a half minutes is not much training data. The first model I trained on it did not work.
The from-scratch model failed
The first attempt was a small VITS trained from scratch. It ran for 800 epochs and still sounded like a robot. I suspect two causes, and I did not isolate them: 6.5 minutes is too little to learn Japanese from scratch, and GAN training is fragile at this data size. I changed the plan: start from a model that already speaks Japanese, then teach it my voice.
The teacher was small but slow
The teacher started as ja_JP-hi_fi_captain-medium, a pretrained Japanese voice from Piper. I fine-tuned it on my recordings. The recordings were enough to move the pretrained voice toward mine; to my ear, the result sounded like me rather than the base voice. I exported it as a 22.8MB int8 ONNX file. In the browser, through ONNX Runtime Web and WASM, a fresh check in Chrome 155 on my M1 Pro CPU took 4.32 to 4.64 seconds per synthesis across five runs of the demo’s default sentence. Model loading is excluded.
At 22.8MB, the exported teacher was already small for a TTS model. At over four seconds for this short sentence, it was too slow for an interactive page. A smaller model would cut both compute time and download size. Medium is Piper's quality label. Small and XSmall are my labels for the two distilled students. All three use int8 quantization, although some layers remain in floating point. File size depends on both the model configuration and which layers are quantized.
The XSmall student is 5.3MB
The XSmall student is a 5.3MB ONNX file using int8 quantization, about the size of one phone photo. It learned by imitating 3,023 teacher-generated utterances. The teacher produces the target; the student learns to produce the same sound from the same phoneme sequence.
The smaller VITS configuration is the main reason the student is 5.3MB instead of 22.8MB. Distillation gave the smaller configuration 149.7 minutes of consistent targets instead of 6.5 minutes of recordings. I did not use int4: ONNX Runtime's quantization guide says its 4-bit kernels target MatMul (matrix multiplication), and most of a VITS model is Conv and ConvTranspose.
In Chrome 155 on the same M1 Pro, the 5.3MB student synthesized the demo’s default sentence in 0.65 to 0.74 seconds across five runs with the model already loaded. That made the page practical.
What VITS is
VITS is a neural text-to-speech architecture. It takes a sequence of sound units and produces a waveform in one model, without a separate vocoder. The name comes from the VITS paper.
Inside VITS, a text encoder predicts a distribution over hidden states, a duration predictor estimates how long each phoneme lasts, and a decoder turns the result into a waveform. At inference, only the generator ships to the browser. That is one reason the exported teacher is far smaller than the 885MB training checkpoint behind it.
Explore the VITS architecture
Inference
ONNX export- Block 1 to 4MHSA + FFN
- Each blockLayerNorm + residual
- Coupling 1 to 4shift + flip
- Upsample×8, ×8, ×4
- MRFk = 3, 5, 7
- Dilations[1,2] [2,6] [3,12]
Training
not exportedSelect a block
Each block in the diagram has a job. Select one to see what it does and why it is there.
Japanese text needs its own converter
The TTS model does not read kanji. It expects phoneme IDs. OpenJTalk can synthesize speech with its own bundled voice, but here I use only its text-to-phoneme frontend. On a server, pyopenjtalk would run that step; on a static page, I had to port it to JavaScript.
A phoneme is the smallest unit of sound that changes meaning. Japanese か is k + a, and し is sh + i. The G2P layer turns kanji and kana into those units. The model never sees letters.
The browser converter reads hiragana, katakana, numbers, Latin letters, and common kanji. OpenJTalk marks where pitch rises and falls; the browser converter does not. I added a hand-written pitch pattern for one word, so almost every other word reaches the model without a pitch pattern. Kanji outside its dictionary are skipped. Latin letters are converted one at a time, so English text is not pronounced as English.
Hear the difference
All three clips use the same sentence, phoneme IDs, and synthesis scales: [0.4, 1.15, 0.4] (noise, length, and duration noise). I pre-generated the files so they do not depend on your CPU or network, and normalized them to RMS 0.08 so loudness does not bias the comparison.
Listen for how far the pitch moves toward the end of each clip.
雨の日の匂いは、遠い国の記憶みたいだ。
The smell of a rainy day is like a memory from a distant country.
Pre-generated audio. No model files are downloaded for these clips. The comparison uses Piper's phonemizer; the live demo uses the browser G2P, so timing and pitch accent can differ.
The words survived. The expression did not.
I wrote five Japanese sentences and checked them against the training metadata and the sources behind the 3,023-utterance corpus, so none appeared in my fine-tuning or distillation data. The comparison above uses one of them.
I synthesized the five sentences with Piper's phonemizer, so the browser G2P was out of the loop. To my ear, the 5.3MB model pronounced all five correctly and kept the teacher's timbre. The problem was prosody: the pitch stayed in a narrow band, the end of a phrase often sounded like the start, and the sentence had no shape.
Two suspects remain: capacity and quantization. The live demo adds a third problem: the browser G2P does not reproduce OpenJTalk's accent marks.
I trained an 11.7MB int8 student to test the capacity explanation. To my ear, its prosody improved, which points to capacity. I still ship the 5.3MB one because the smaller file is the point of the page.
Try a sentence
The page loads the demo code. Pressing Speak downloads the model, the WASM runtime, and the G2P data, then creates the ONNX session. Later runs on the same page reuse the loaded model. The timing panel separates model load, generation, and audio length, then shows how much faster than real time the voice was generated.
Demo
Hear the cloned voice
One 5.3MB voice, reading Japanese in the browser. No inference server.
Common examples use common kanji; Jugemu stays in kana. The English-ish preset feeds katakana syllables to the Japanese model, so it is an accent experiment. It does not add an English mode. Latin text is still spelled one letter at a time.
The 5.3MB voice loads when you press Speak.
5.3MB model weights. ONNX runtime adds about 14MB. This is a clone of my voice, trained ahead of time. It does not clone your voice. Unknown kanji are skipped, and most pitch accent is missing.
What the page loads
The page is static. The demo needs a 5.3MB ONNX model, a 14MB WASM runtime, and about 1MB of G2P data. The first run is closer to 20MB. After that, the runtime can be cached, and the model stays in memory for as long as the page is open.
The text never leaves the browser: G2P, synthesis, and playback all run on the page.
A zero-shot clone is still too large
The next experiment is a zero-shot cloner small enough for a static page. I tested CosyVoice 3 with an 8.3-second reference. To my ear, the samples sounded like me. But the 0.5B release takes about 9.1GB on disk and ran about 4 to 6 times slower than real time (RTF 4 to 6) on my M1 Pro CPU. That belongs on a desktop or server. If a cloner fits in tens of megabytes and runs near real time on a CPU, it could move into a page like this one.
Technical notes
The training machine was a MacBook Pro 18,3: Apple M1 Pro, 16GB RAM, macOS 27.0.1. The run used Python 3.11.4, PyTorch 2.14.1 with MPS, Lightning 2.6.6, piper-tts 1.8.0, and onnxruntime 1.31.0. The teacher fine-tune used fp32, batch 4, and about 62 seconds per epoch.
- Teacher checkpoint
- ~885MB
- Teacher browser export
- 22.8MB int8 ONNX
- Student browser export
- 5.3MB int8 ONNX
- G2P dictionary
- 34,176 dictionary entries · 330 kana entries
The 5.3MB model is shared under CC BY-NC-SA 4.0: credit the source, use it for noncommercial purposes, and share adaptations under the same license. The 22.8MB teacher model is not distributed. See the credits and changes for the base voice and training data.
ONNX Runtime Web has its own license and third-party notices.
- Piper: the VITS training and inference toolkit. The base voice was ja_JP-hi_fi_captain-medium.
- VITS paper: the model family behind the voice.
- ONNX Runtime Web and ONNX: the browser runtime and model format.
- pyopenjtalk and OpenJTalk: Japanese text to phonemes.
- PyTorch, Lightning, and librosa: training, checkpoints, and audio loading.