Fine-Grained Style Transfer TTS

A lightweight (48M-parameter) zero-shot style-transfer TTS system that disentangles speaker from style, reaching 3.64 prosody pMOS against much larger baselines.

JASA Express Letters 6(3), 034802 · 2026 Eray Eren, Q. Liu, A. Alwan, G. Bharaj
DOI Publisher Original demo page
3.64prosody similarity MOS
+31%relative style preference
48Mparameters
3 baselinesGenerSpeech, YourTTS, VALL-E-X

Zero-shot style transfer asks a TTS system to speak new text in the style of a reference clip it has never heard before. Two things make that hard: the models that do it well tend to be very large, and fine-grained style, meaning the pitch and duration habits within a phrase, often fails to carry over, so the output ends up in the right voice but the wrong manner.

This work uses a small GenerSpeech backbone plus a fine-grained style encoder, and adds two losses to pull the representations apart: a mutual-information minimisation loss separating speaker, global style, fine-grained style and content; and a maximum-mean-discrepancy-guided cycle-consistency loss that further detaches style from speaker identity and widens the range of style embeddings.

On VCTK it improves average style preference by a relative 31% over GenerSpeech, YourTTS and VALL-E-X, with a prosody-similarity MOS of 3.64.

Overall model architecture.
Model architecture.
Fine-grained style encoder with the disentanglement losses.
Style encoder and disentanglement losses.

Listen

In every row the system is given the reference clip and the target text, and must produce that text in the reference speaker’s style. Ground Truth is the real recording of the target sentence.

What to listen for: whether the pace, emphasis and pitch range track the reference, not just whether the voice timbre matches. The common failure in the baselines is a style mismatch: correct-sounding speaker, but flatter or differently paced delivery than the reference.

In-domain: trained on VCTK, tested on VCTK

Utterance Reference style is transferred from this speaker Ground Truth Ours GenerSpeech YourTTS VALL-E-X
1 And we will go meet her wednesday at the train station.
2 His friends say he's looking for the pot of gold at the end of the rainbow.
3 But refraction by the raindrops which causes the rainbows.
4 And the width of the colored band increases as the size of the drops increases.
5 The result is to give a bow with an abnormally wide yellow band, since red and green light when mixed form yellow.
6 And maybe a snack for her brother Bob.

Out-of-domain: trained on LibriTTS, tested on VCTK

The harder setting: the model never saw VCTK during training, so both the speaker and the recording conditions are new.

Utterance Reference style is transferred from this speaker Ground Truth Ours GenerSpeech YourTTS VALL-E-X
1 And maybe a snack for her brother Bob.
2 And its two ends apparently beyond the horizon.
3 His friends say he is looking for the pot of gold at the end of the rainbow.
4 Wednesday at the train station.
5 But refraction by the raindrops which causes the rainbows.
6 She would never resort to such devices.

Scroll the tables sideways to reach every system; the utterance column stays pinned. Starting one clip pauses the others.

Citation

@article{eren2026styletransfer,
  title   = {Improving zero-shot style transfer text-to-speech by disentangled fine-grained style modeling},
  author  = {Eren, Eray and Liu, Qingju and Alwan, Abeer and Bharaj, Gaurav},
  journal = {JASA Express Letters},
  volume  = {6},
  number  = {3},
  pages   = {034802},
  year    = {2026},
  doi     = {10.1121/10.0042974}
}