How to Make an AI Voice Whisper Naturally for ASMR

It is two in the morning. Your character, a soft-spoken moth tailor named Sable, is supposed to lean in and murmur, “Hold this end for me. Gently. Like it’s asleep.” You type the line into a text-to-speech tool, add whispering to the prompt, and press generate.

What comes back is a newsreader who has been asked to keep it down. A faint hum sits under every vowel, a breath lands in the middle of “gently” like a small gasp, and the pause before “Like it’s asleep” has moved to after “Like.” Version nine, eleven tries later, moves the breath somewhere better and the hum somewhere worse. Nothing is broken. You have asked a speech model for the kind of speaking it was least built to do.

1. Why a whisper breaks text-to-speech

A whisper has no buzz

Normal speech has two parts. Your vocal folds, two small flaps of tissue in the larynx, snap open and shut many times a second and make a buzz; the speed of that buzz is the pitch you hear, its fundamental frequency. Your throat and mouth shape the buzz into vowels by boosting some frequencies; those peaks are formants, the same ones behind the chipmunk effect in our guide to building a consistent VTuber voice.

A whisper removes the buzz. The vocal folds are held apart, and air rushing through the gap makes a hiss of turbulence instead. Phoneticians call this unvoiced speech. In talking, only sounds like the “s” in sit are; in a whisper, every sound is. The mouth still shapes the hiss, so words stay clear, but researchers report that whispered vowels have their first two formants shifted upward, last longer and carry much less energy. Amazon researchers who built Alexa’s whisper mode add that consonants are stronger relative to the rest, so turning a whisper up brings the hiss up with it.

Models are built around the buzz

Text-to-speech systems learn mostly from normal talking, and many end with a vocoder, the stage that turns the model’s description of sound into a waveform, often guided by a pitch track. A March 2026 paper on WhispSynth, a synthetic whisper collection, spells out the result. Speech collections and vocoders “are predominantly optimized for modal phonation” (ordinary voiced speech), and state-of-the-art systems asked to whisper “tend to produce residual pitch.” The speaker encoder, the part that listens to a reference clip to decide which voice to copy, “frequently misconstrue[s] whisper prompts as hoarseness or environmental noise.”

That leftover buzz is the most plausible source of the hum under Sable’s vowels, and of what creators call a “metallic” whisper (the paper does not use the word). Breaths go wrong too: a text model puts them where narration would, at punctuation, but a whisper uses more air and runs out sooner, so real whisperers breathe elsewhere.

Quiet is not the same as whispered

Plenty of ASMR is soft-spoken, which most TTS voices already do; if your character only needs to be gentle, stop here and save yourself a week. If she must whisper, the stakes are real: in an early peer-reviewed ASMR study, Emma Barratt and Nick Davis (2015) found whispering was the most common trigger (75% of 475 participants), and 82% used ASMR to help them sleep. A buzz is a bad flaw in something people hear with their eyes closed.

2. Three routes to a synthetic whisper

Route 1: text-to-speech with prompts and settings

A model can only whisper if it learned from whispers and gives you a way to ask. Open models fall into four groups:

• No whisper control. Resemble AI’s Chatterbox (MIT licence) has exaggeration and cfg_weight settings for drama and pace, and its Turbo variant adds tags such as [laugh]. Nothing for whispering.
• Breath sounds, no whisper. Nari Labs’ Dia (Apache 2.0) has (inhales) and (sighs) tags; Canopy Labs’ Orpheus (Apache 2.0) has <sigh> and <yawn>. These place breaths but do not switch off the buzz.
• A documented whisper style. Hugging Face’s Parler-TTS Mini: Expresso (Apache 2.0) accepts “whisper” in its plain-English style prompt, but only in the four studio voices of Meta’s Expresso dataset, whose licence is non-commercial; read both licences before a monetised channel relies on it. Fish Audio’s Fish Speech documents a [whisper] tag under the company’s own community licence.
• A whisper edit. StepFun’s Step-Audio-EditX (Apache 2.0) works on a recording rather than on text: give it a take and the instruction “whisper,” and it re-voices the take as a whisper. Each pass removes more of the buzz, so the edit can be run again on its own output.

A slider that makes a voice calmer or slower will not invent unvoiced speech.

Route 2: audio tags in a commercial model

Audio tags are short instructions in square brackets, written into the text, that tell the model how to deliver what follows. ElevenLabs’ Eleven v3 and v4 accept [whispers], [hushed], [barely audible], [pause], [sighs] and tags you write yourself, such as [whispering, fearful].

Its prompting guide has two catches. First: “The voice still matters. A delivery that is already in the training data — whispering, shouting, a particular delivery — is easier for the model to reproduce.” ElevenLabs says v4 “follows audio tags more reliably than earlier models, including when the voice wasn’t trained on that delivery,” but still tells you to test your exact tags with your exact voice. Second, v3 and v4 do not support the SSML break tag (Speech Synthesis Markup Language, an older standard for “pause here for 0.8 seconds”). You shape pauses with punctuation, ellipses and [pause]: you describe a pause rather than place it.

Route 3: perform the whisper, then convert it

Speech-to-speech, or voice conversion, re-voices a recording as a different speaker but keeps its timing and breathing. A human has already produced real unvoiced speech with breaths in the right places; the model only changes who is whispering.

ElevenLabs says its Voice Changer preserves “whispers, laughs, cries, accents, and subtle emotional cues” and claims “greater accuracy with whispering.” It converts uploaded segments of up to five minutes, not live, and can remove background noise as it goes.

Open-source converters need a test first. Many, including the RVC family behind most community voice models, rebuild the voice around a pitch track from your recording, and a whisper has none. RVC’s “protect” setting, which shields “voiceless consonants and breath sounds” from artifacts, shows unvoiced sound is fragile there; nothing in its documentation covers a fully whispered take. Seed-VC (GPL-3.0; archived in November 2025) documents pitch conditioning only for its singing model. Whispering live into a real-time changer adds delay to that risk; see ElevenLabs versus Voicemod and voice changer latency.

The three routes side by side

We built Sable’s shop from open-weight models and generated sounds only. Her two lines started as normal speech from Chatterbox and were whispered with Step-Audio-EditX; the five shop sounds are synthesised from filtered noise, with no recordings. Put on headphones and move things around your head: this is the binaural placement described in section 3, done by your browser.

What usually separates a natural whisper from an uncanny one is not the voice. It is where the air goes.

3. Recording, writing and cleaning a whisper

Route 3 moves the hard part into your room, and a converter keeps whatever you give it, including the fridge.

Breath, distance and level

• Breathe before the line, not in it. A separate breath can be kept, shortened or cut; one that runs into the first word cannot.
• Keep breath, level and distance the same all session. A converter preserves how you breathe, so a gasping first take and a relaxed tenth sound like two characters, and a whisper that drifts from three centimetres to fifteen sounds as if she walked away.

Close means a few centimetres. Directional microphones (such as the common heart-shaped “cardioid” type) boost the bass at that range; this proximity effect can turn every breath into a rumble, so move back slightly or cut the lowest frequencies later.

Do not fix a quiet whisper with gain alone: it also raises the microphone’s self-noise, the faint hiss it makes in a silent room. LEWITT, a microphone maker, says “whispered vocals, ASMR” call for “ultra-low self-noise,” and ElevenLabs warns that “a quiet recording may hinder AI recognition.” Get closer first. A whisper is mostly air, so plosives, the small blasts on “p” and “b,” hit the capsule hard (“it sounds like an explosion, indeed,” says Neumann). Use a pop screen, angle the microphone slightly off your mouth, and record ten seconds of room silence each session for noise reduction to learn from.

Write lines, not narration

• One breath, one action. Sable says “Hold this end for me,” then waits as if the listener is taking the scarf. On routes 1 and 2 short lines also leave the model less room to misplace a breath.
• Second person, present tense. Personal attention was the second most common trigger in the Barratt and Davis survey; “Let me see it” is attention, “Sable inspected the scarf” is a story.
• Few hissing sounds. Whispered consonants are already loud, so “she sews six silk sashes” becomes a hiss with a vowel somewhere inside it. A whispered “fourteen” and “forty” are easy to confuse, too.

Clean gently, then build the space

Mouth sounds, the clicks and lip smacks a close microphone catches, trigger some listeners and drive others away; pick one policy per channel and use a dedicated tool such as iZotope RX’s Mouth De-click, designed for “clicks and lip smacks.” Treat breaths the same way: keep the ones that pace the scene, remove any mid-word, and never silence them all, or the whisper has no body behind it.

Noise reduction pushed too far eats the whisper along with the hiss. iZotope warns that “noise reduction artifacts can be more distracting than the noise itself” and recommends several gentle passes over one heavy one. Stop when the background is unnoticeable, not gone. A de-esser, a compressor that reacts only to harsh “s” and “sh” sounds (sibilance, roughly 4–10 kHz), usually needs more work on a whisper than on speech; too much gives the character a lisp. Write the order down and keep it: trim, mouth clicks, noise reduction, de-essing, level.

None of the voice tools covered here claims spatial output, so a generated whisper is a single track with no position. Binaural audio is stereo that places each sound around your head on headphones (over speakers the illusion collapses). It is recorded with a dummy head such as the Neumann KU 100, a microphone in each ear, or built in the mix with HRTF (head-related transfer function) filters measured from real heads, which binaural panner plugins use to place a mono voice anywhere. TH Köln publishes free KU 100 measurements under a Creative Commons licence.

4. One scene, start to finish

Sable keeps her mending shop open all night, because customers only come when they cannot sleep. Here is a three-minute scene on route 3.

1. Write six to ten lines, from “Oh. You’re still up. Come in, mind the step” to “All finished. Keep it close tonight.”
2. Fill in the voice sheet. True whisper, not soft-spoken; about four centimetres, slightly off-axis; breathe before each line, never inside one. Add the target voice (designed, licensed or your own clone) with tool, model name and voice ID.
3. Record dry, three takes per line, no effects.
4. Convert one line first and listen on headphones for hum, changed breaths or harsh consonants. If pitch creeps in, try the background-noise option, a slightly louder (still unvoiced) whisper, or another converter.
5. Convert the rest and assemble, with pauses longer than feels natural in the editor; the listener is imagining the action.
6. Clean in the fixed order, saving each stage as a file the first time so you can find which step caused a problem.
7. Place it in space. Move Sable to one ear for “Hold this end for me” and back to centre for the last line, and put foley (small recorded sounds such as fabric or thread) on its own track where the action happens. The booth in section 2 lets you try this before you mix.
8. Label it as AI-converted; the consistent-voice guide covers platform disclosure rules.

On route 2, steps 3 to 5 become: pick a voice whose sample already whispers, tag each line [whispers] with ellipses for the pauses, and keep the best of several takes.

Whose whisper is it?

In ASMR the voice is the product. Never clone or convert into a real person’s voice without written permission. ElevenLabs’ usage policy (updated 9 October 2026) prohibits “cloning, synthesizing, or impersonating the voice, identity, or likeness of any living individual without their explicit and informed consent.” The law is moving the same way. Tennessee’s ELVIS Act (in force since July 2024) protects voices against AI simulation, including from people who knowingly distribute tools built mainly for unauthorised replicas. The US NO FAKES Act cleared the Senate Judiciary Committee in June 2026 but was stalled as of early October. Since 2 August 2026, the EU AI Act’s transparency rules cover AI audio that resembles real people and would falsely appear authentic, with a lighter duty for clearly fictional work.

Your own dry whispers are exactly what someone would need to clone you, so keep them offline and publish only the converted scene. With a performer, get written permission for the specific character, channel and conversion.

So, can an AI whisper naturally?

Yes, if natural means nothing pulls the listener out: no hum under the vowels, no gasp mid-word, no pause in the wrong place. Today that comes most reliably from a person’s breath and timing, re-voiced by speech-to-speech. Sable is fictional and nobody has cloned her. Keep it that way.

Sources

• Wikipedia, “Whispering”: vocal folds held apart, turbulence, unvoiced phonation (citing Laver, Principles of Phonetics). Checked 19 Oct 2026.
• M. Cotescu, T. Drugman, G. Huybrechts, J. Lorenzo-Trueba, A. Moinet, “Voice Conversion for Whispered Speech Synthesis,” IEEE Signal Processing Letters (arXiv:1912.05289): acoustic differences of whispered speech; Alexa Whisper Mode.
• T. Tan et al., “WhispSynth: Scaling Multilingual Whisper Corpus through Real Data Curation and A Novel Pitch-free Generative Framework,” arXiv:2603.14853, March 2026: bias toward ordinary voiced speech; residual pitch in TTS whispers.
• “State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition,” arXiv:2506.16969: summary of whisper acoustics (formant shifts, energy).
• E. Barratt and N. Davis, “Autonomous Sensory Meridian Response (ASMR): a flow-like mental state,” PeerJ 3:e851, 2015.
• ElevenLabs documentation: Best practices (prompting); audio tags list (blog, updated 15 Oct 2026); Voice Changer (capabilities page and product guide); Usage Policy (updated 9 Oct 2026). Checked 19 Oct 2026.
• Resemble AI, Chatterbox GitHub repository and Chatterbox-Turbo model card. Checked 19 Oct 2026.
• Hugging Face, Parler-TTS Mini: Expresso model card; Meta AI, Expresso dataset page. Checked 19 Oct 2026.
• GitHub repositories for Nari Labs Dia, Canopy Labs Orpheus TTS and Fish Audio Fish Speech. Checked 19 Oct 2026.
• RVC-Project, Retrieval-based-Voice-Conversion-WebUI interface strings; Plachtaa, Seed-VC GitHub repository. Checked 19 Oct 2026.
• LEWITT, “Microphone self-noise”; Neumann, “How to Protect Your Microphone Against ‘Pops’”; Wikipedia, “Proximity effect (audio).”
• iZotope, “Reducing ambient noise from home studio recordings”; “The dos and don’ts of de-essing”; RX Mouth De-click documentation.
• Wikipedia, “Binaural recording”; Neumann KU 100 product page; TH Köln, “Spherical Far-Field HRIR Compilation of the Neumann KU100” (Zenodo, CC BY 3.0).
• RecordingLaw, “Deepfake & AI Voice Cloning Laws by State (2026)” (ELVIS Act); Daily Journal, “NO FAKES Act dead for now as Cruz objects to unanimous consent request,” 7 Oct 2026; Future of Life Institute, “The EU AI Act’s transparency rules: a practical guide to Article 50.”