Beyond Naturalness

EMNLP 2026 · ServiceNow

Beyond Naturalness

Probing automated text-to-speech evaluators on linguistically grounded dimensions

Synthesized speech is usually scored for “naturalness”, a concept that is ill-defined and, without further elaboration, uninformative. We asked whether automated evaluators detect the specific errors that listeners hear. We split naturalness into ten dimensions at three levels and made clips that each contain one error. Trained linguists labelled every clip, and we checked which errors the evaluators detect.

MOS predictors detect errors that sound wrong in any sentence, such as glitches or the wrong pace, and miss errors that are only wrong given the words, such as a mispronounced word. Audio-LLM judges are inconsistent: the Gemini models detect some word-level errors, depending on the prompt, and the open-weight models detect few errors of any kind.

860
clips
10
linguistically grounded dimensions
8,600
majority-voted human labels
8
automated evaluators

Listen to the clips Paper on arXiv ↗ Code and data ↗

Hear one error

A rag will soak up spilled water.

/s/ voiceless, as written[z] voiced: the vocal folds vibrate

Listen

The ten dimensions are grouped into three levels: the individual word, the sentence as it is spoken, and the speaker’s voice. For each dimension, play a reference clip, then the same sentence with one error added.

Text what is said

Riceisoftenservedinroundbowls.

Three levels

Word each word

ɹaɪsɪzˈɔ.fənsɝvdɪnɹaʊndboʊlz

Prosodic the sentence

Paralinguistic the voice

female, about 35stays with the speaker

neutral emotion · natural rangefollows the content

One sentence seen at each level. The text itself is not a level; the levels describe how it is spoken. Word-level dimensions judge each word on its own, and prosodic dimensions judge how the whole sentence is spoken. Paralinguistic dimensions judge what the voice conveys about the speaker rather than the message: some traits stay with the speaker (identity, and whether a human could produce the sound), while others follow the content (emotion and expressiveness).

Level 1 of 3 · 2 dimensions

Word level

Does each word sound like the right word? These properties are listed with the word in a dictionary: which sounds it is made of, and which syllable is stressed.

Phonetic accuracy

Each word is made of the right sounds. A vowel or consonant swapped for another counts as an error, and so does the wrong pronunciation of a word that has two (lead the metal, lead the way).

How it differs from neighbouring dimensions

Listeners recognize words by their sounds, so one wrong sound can turn a word into a different word or a non-word: “soak” said as “zoak”, or “branches” as “breenches”. It is the most basic way TTS can fail, and it can happen in a clip whose melody, timing and voice are all fine, so it needs its own check.

Wrong vowel

Hang tinsel from both branches.

“branches” has the vowel of “bean” instead of “ban”: /ˈbræntʃɪz/ is heard as [ˈbrintʃɪz].

Wrong consonant

A rag will soak up spilled water.

“soak” starts with a voiced [z] instead of /s/, so it sounds like “zoak”.

Lexical stress

Each word has its stress on the right syllable, the one a dictionary marks: YEL-low, not yel-LOW.

How it differs from neighbouring dimensions

Stress is part of how a word is recognized. Moving it can make a familiar word hard to catch, or turn it into another word: RE-cord (the noun) and re-CORD (the verb) differ in stress, and the vowels change with it. This is about syllables inside one word; which words in the sentence get emphasis is prosodic stress.

Stress on the wrong syllable

Leaves turn brown and yellow in the fall.

“yellow” is stressed on the second syllable, [jəˈloʊ], instead of the first, /ˈjɛloʊ/. Without stress, the first vowel weakens to a schwa.

Level 2 of 3 · 4 dimensions

Prosodic level

Does the way the sentence is spoken carry its structure and meaning? This covers melody, emphasis, phrasing and pace.

Intonation

The melody of the sentence: how pitch rises and falls across it. Yes/no questions often end on a rise. Statements and most wh-questions end on a fall.

How it differs from neighbouring dimensions

Pitch tells the listener what kind of sentence they are hearing and whether the speaker has finished. A question with a flattened rise can come across as a statement, and pitch that wobbles or wanders sounds wrong even when every word is right. Intonation is the shape of the pitch across the whole sentence. Which words stand out is prosodic stress.

Pitch wobble

Does anyone here remember what the old procedure used to be?

The pitch wavers up and down several times a second, so the voice sounds unsteady.

Pitch range too wide

What is Data Informed Diplomacy, with all its various manifestations?

Pitch swings far wider than normal across the question. Timing and loudness match the reference, so only the melody is wrong.

Prosodic stress

Which words in the sentence get extra emphasis, through pitch, loudness and length. Normally the emphasis lands on the words that carry the meaning, not on small words like “were” or “in”.

How it differs from neighbouring dimensions

Emphasis points the listener at the important part of the sentence. “I didn’t take your car” means something different depending on which word is stressed, so emphasis in the wrong place changes the meaning or makes the speaker sound strange.

Stress on small words

Ten pins were set in order.

“were” and “in” are emphasized, while the words that carry the meaning are not.

Prosodic boundary placement

How the speaker groups words into phrases, marked by short pauses, slowing down at the end of a phrase, and changes in pitch.

How it differs from neighbouring dimensions

Grouping tells the listener which words belong together. Leave out a needed break and “When the dog bites, the man screams” is heard as “the dog bites the man”, which stops making sense at “screams”. Add a break where none belongs and words that go together get split (“they quickly made, up a clever excuse”), which sounds choppy.

Natural phrasing (different sentence)

Children play in the park every afternoon.

Missing break

When the dog bites the man screams for help.

No break after “bites”, so “the man” sounds like the object of “bites”, and the sentence stops making sense at “screams”.

Break inside a phrase

They quickly made up a clever excuse.

A break splits “made” from “up”, two words that belong together.

No break between clauses

The rain stopped so we went for a walk.

No break before “so”, so the two clauses run together. The meaning survives, but it sounds rushed.

Speech rate appropriateness

How fast the speech is overall, and whether that pace suits the content.

How it differs from neighbouring dimensions

Speech that is too fast is hard to follow, and speech that is too slow sounds sluggish. A system can get every sound, emphasis and melody right and still talk at the wrong speed, so pace is checked on its own.

Very fast

Men strive but seldom get rich.

The same sentence in about 60% of the reference’s time.

Very slow

The first worm gets snapped early.

Nearly twice as long as the reference.

Level 3 of 3 · 4 dimensions

Paralinguistic level

Does the voice suit the speaker and the content? This covers emotion, how animated the voice is, whether it stays one person, and whether a human could have produced it.

Emotional appropriateness

The emotion in the voice matches what is being said. News of a death should not sound cheerful, and good news should not sound bored.

How it differs from neighbouring dimensions

Listeners notice the wrong emotion immediately, and it changes how the message lands: a condolence read cheerfully sounds insensitive. This dimension asks only whether the kind of emotion is right. How much emotion there is belongs to expressiveness.

Wrong emotion

“We’re going to Paris, I’m so thrilled!” she squealed with delight.

Read in a calm, neutral voice, while the words and “squealed with delight” call for excitement.

Expressiveness

How much the voice varies in pitch, loudness and pace, judged against what the content calls for. Flat, robotic delivery and over-the-top, theatrical delivery both count as errors.

How it differs from neighbouring dimensions

A clip can carry the right emotion and still sound monotone, or sound like it is overacting. Neither sounds like a person talking. Rating the amount of variation separately from the kind of emotion lets us tell those two failures apart.

The three clips use different sentences and voices. Compare how much the voice moves, not which voice it is.

Monotone

Open your book to the first page.

Pitch and loudness barely move, so the sentence sounds flat.

Natural

Many hands help get the job done.

Overacted

Fish evade the net and swim off.

Broadcast-style swings in pitch and loudness on an everyday sentence.

Speaker identity consistency

The voice stays one person throughout: the same pitch range, voice quality, apparent age, gender and accent, within a sentence and across sentences.

How it differs from neighbouring dimensions

Every moment of a clip can sound like a believable person while the whole clip is two people stitched together. Listeners hear that as a change of speaker, and a check that looks at one moment at a time would miss it.

Voice changes mid-word

Screw the round cap on as tight as needed.

Partway through “cap” the voice fades into a different man’s voice.

New voice for the second sentence

Paper will dry out when wet. A gold vase is both rare and costly.

The second sentence is read by a different woman.

Human plausibility

A human could physically have produced the sound. Errors are digital artifacts: buzzing, clicks, metallic tones, impossible jumps in pitch, or syllables cut off.

How it differs from neighbouring dimensions

Glitches break the impression of a real voice even when the words, emphasis and emotion are all right. They come from the audio signal rather than the language, so we keep them apart from errors like a mispronounced word, which still sounds human.

Dropouts and hiss

Nudge gently but wake her now.

“gently” breaks up into fragments, and a faint hiss runs under the whole sentence.

Distorted consonants

It takes heat to bring out the odor.

The /t/ and /k/ in “takes” sound distorted, as if the microphone were overloaded. The rest is clean.

How the benchmark was built

How the benchmark was built Pipeline: sentences from EmergentTTS-Eval, Harvard Sentences and newly written ones get one dimension-specific error each, are synthesized with TTS or edited acoustically, labelled by three linguists on every dimension, then scored by MOS predictors and Audio-LLM judges and correlated with the human labels per dimension. Data Generation Source Corpora EmergentTTS-Eval Harvard Sentences Original Synthetically Generated abWord Prosodic Para-linguistic “Dull stories make her laugh” “Type out three lists of orders” “My beloved dog passed away quietly in his sleep tonight” 3 Levels 10 Dims Input Sentences Dimension Specific Error Injection Apply dimension-specific modifications to input text LLM-altered IPA laugh→/ˈlæf/→/ˈlʊf/ LLM-altered text Wrap preposition in quotes to inject stress API-enforced emotion Apply ‘euphoric’ tag “Dull stories make her <<ˈ|l|ʊ|f>>” “Type out three lists “of” orders” “[emotion: euphoric] My beloved dog passed away quietly in his sleep tonight.” Output Sentences TTS AcousticManipulation Linguist Annotation Score every dimension for each speech sample: failure (0) good (1) 860 Samples Majority Voting Evaluation Automated Judge Classes MOS Predictors Audio LLM Judges Dimension Level Correlations Word Prosodic Paralinguistic 0.12 0.23 0.4 0.32 0.36 0.15 Input Conditions Transcript Access Dimension description access Output granularity Metrics Correlations AUROC
Correlations shown are illustrative; measured results are under Findings.Paper figure ↗

Click a level to show only that level · Esc to close
  1. Pick sentencesFrom Harvard Sentences and EmergentTTS-Eval, plus new ones where a dimension needed them.
  2. Add one errorAlter the pronunciation, punctuation or requested emotion before synthesis (Cartesia Sonic-3, ElevenLabs Multilingual v2), or edit the audio after it.
  3. Label by earThree linguists mark error or no error on all ten dimensions. An error counts only if at least two of the three hear it, so the label is what they heard, not what we injected.
  4. Score the evaluatorsFour MOS predictors and four Audio-LLM judges score the same 860 clips. Kendall’s τb measures how well each follows the linguists, dimension by dimension.

How the evaluators were asked

MOS predictors only hear the audio. Audio-LLM judges also read a prompt, and what they detect depends on it, so we asked them in four ways, each with and without the transcript.

MOS predictors

audio → a score per output head

Audiobox Aesthetics (4 output heads), DNSMOS Pro (2), NISQA and UTMOSv2. Each head returns a continuous score and is tested on its own.

Audio-LLM judges

audio + prompt, ± transcript → 0 error · 1 no error

Gemini 3 Flash and Gemini 3.5 Flash through the Google API; Qwen3-Omni and Step-Audio-2-Mini, open-weight. Each judge answers three times per clip, and the majority answer counts.

  1. C1NaturalnessRate the overall naturalness, with no definitions. The closest a judge gets to a MOS predictor.defines none · 1 score
  2. C2aDimensions definedThe prompt defines all ten dimensions, but the judge still gives one overall score.defines 10 · 1 score
  3. C2bAll ten in one callThe same definitions, and the judge scores each of the ten dimensions in one reply.defines 10 · 10 scores
  4. C3One at a timeOne call per dimension, with only that dimension’s definition.defines 1 · 1 score · 10 calls

The prompts define the same ten dimensions as Listen. The full prompt text is in Appendix 6.7 of the paper ↗

What we found

None of the eight evaluators detects every kind of error the linguists hear.

EvaluatorWordProsodicParalinguistic
PALSINPSPBSREAEXSIHP
MOS predictors Each output head is scored as its own evaluator.Audiobox Aesthetics4 headscontent enjoyment, content usefulness, production complexity, production qualityDNSMOS Pro2 headstrained on BVCC; trained on VCC2018NISQA1 headoverall quality scoreUTMOSv21 headoverall quality score0032050607
Gemini 3 Flashdetecteddetectednot detectednot detectednot detectednot detectednot detecteddetectednot detectednot detected
Gemini 3.5 Flashdetecteddetectednot detecteddetecteddetectednot detectednot detectednot detecteddetectednot detected
Gemini 3.8 Flash newdetecteddetectednot detecteddetecteddetecteddetectednot detectednot detecteddetectednot detected
Qwen3-Omninot detectednot detectednot detectednot detectednot detecteddetectednot detectednot detecteddetectednot detected
Step-Audio-2-Mininot detectednot detectednot detectednot detectednot detectednot detectednot detectednot detectednot detectednot detected
A filled cell means a significant positive Kendall’s τb with the linguists’ labels (p < .05). For MOS predictors the number is how many of the 8 output heads reach it. Significant negative correlations are not shown. Select a column to hear that dimension.
  1. MOS predictors detect errors that sound wrong whatever the sentence says.They catch glitches, wrong pace and expressiveness, and miss errors that depend on the words, such as mispronunciations.
  2. Gemini judges detect word-level errors, depending on the prompt.The open-weight judges detect few errors under any prompt.
  3. Describing the dimensions in the prompt helps some of them.Given the definitions, Gemini and Qwen3-Omni detect word-level errors even without the transcript.
  4. Asking for all ten scores in one call backfires.Two of the four judges give nearly every clip the top score.
  5. Gemini 3.8 Flash detects the most.Six of the ten dimensions under the naturalness prompt, word-level errors even without the transcript, and seven when asked one dimension at a time. No prompt gets it to detect intonation errors, which three MOS heads catch.

Full results in the paper ↗

The benchmark measures whether an evaluator responds to a known, audible error. It does not rank TTS systems, and the clips are English and mostly short sentences.

Paper

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions ↗

Oluwanifemi Bamgbose*†, Jash Shah†, Simon Rosen†, Lindsay Devon Brin, Hoang H Nguyen, Anke Koelzer, Rachel Hansen, Tara Bogavelli, Fanny Riols

ServiceNow · † equal contribution · * corresponding author, nifemi.bamgbose@servicenow.com

Paper on arXiv ↗Code and data ↗

BibTeX
@inproceedings{bamgbose2026beyond,
  title={Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions},
  author={Bamgbose, Oluwanifemi and Shah, Jash and Rosen, Simon and Brin, Lindsay Devon and Nguyen, Hoang H and Koelzer, Anke and Hansen, Rachel and Bogavelli, Tara and Riols, Fanny},
  booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year={2026},
  url={https://arxiv.org/abs/2608.09930}
}