Voice cloning started as a speech problem. The early production systems (XTTS, ElevenLabs, PlayHT, Resemble) were built on speech datasets and optimized for TTS-style output. Then the market asked for singing. Users wanted to clone their own voice to sing over instrumentals. Artists wanted to generate harmonies in their own style. Developers wanted to build karaoke apps that could produce any song in any voice. The singing use case turned out to be harder and more legally fraught than the speech use case, and the gap in licensed training data became visible fast.
This post is for product and ML teams at voice cloning companies that are trying to add singing capabilities to their platforms. It covers the specific data requirements for singing vs speech, the legal structure needed to deploy at scale, and a practical workflow for integrating licensed singing data into an existing voice cloning pipeline.
Why singing is a different problem from speech
A model that produces convincing spoken output does not automatically produce convincing sung output. The two tasks require different signal characteristics, different training data, and different evaluation criteria.
Pitch control and F0 continuity
Speech has pitch variation, but it is relatively narrow and relatively unconstrained. Singing has pitch variation that is both wider and more structured: specific notes on a specific scale, held for specific durations, with precise transitions between them. A model trained only on speech learns general pitch patterns but does not learn musical pitch. When asked to sing, it produces wandering, out-of-tune vocals.
Singing training data needs to include accurate F0 contours that reflect musical intent. This means pitch tracking that can handle sustained notes, vibrato, pitch bends, and melismas (single syllables spanning multiple notes).
Sustain and dynamic envelope
Spoken words are short and transient. Sung notes are often sustained for seconds, with dynamic envelopes that shape the emotional content (crescendo, decrescendo, swell). Speech-trained models struggle to sustain notes because the training data does not contain enough examples of long, held vowels with stable pitch.
Vowel clarity at extreme pitches
When singers move to high registers, formant structure shifts in ways that are unique to singing. Operatic "chiaroscuro" technique, pop belt, head voice, falsetto — each has a different formant signature, and none of them match the formant patterns in normal speech. A model trained only on speech produces vowels that sound "speechy" even when hitting the right notes.
Vibrato and expression
Vibrato is a controlled oscillation of pitch (typically 5-7 Hz) that singers add for expression. It is a learned technique, not a natural speech pattern. Training data that includes vibrato-rich examples teaches the model to reproduce it; training data without it produces flat, robotic singing.
What licensed singing data adds
A well-constructed singing dataset gives your voice cloning model:
- Wide pitch ranges with accurate F0 tracking across the full vocal range of each singer
- Sustained notes of varying lengths, dynamics, and vowel content
- Vibrato examples across different speeds and depths
- Technique variations including head voice, chest voice, mix, belt, and falsetto
- Articulation at pitch showing how consonants are formed while maintaining a target note
- Emotional dynamics including breathiness, intensity, and expressive microdynamics
Speech datasets do not contain most of these systematically. You can extract some singing-adjacent signals from expressive speech (audiobook narration, acting performances) but you cannot extract musical pitch control from non-musical material. The only way to get singing-specific data is to record singers or to license a singing dataset.
Data requirements for adding singing to a voice cloning product
The exact volume depends on what you are building. Below are three common scenarios and their approximate data needs.
Scenario A: Fine-tune an existing speech voice cloning model for singing
If you already have a speech-trained base model and want to add singing capability as a fine-tune, the data requirement is moderate:
- Recommended: 30 to 100 hours of clean singing data spanning multiple singers, genres, and technique types.
- Minimum viable: 10 to 20 hours if the fine-tune is scoped to a specific singing style (pop only, for example).
- Per-voice fine-tuning on top: 10 to 60 minutes of the target singer's voice for identity cloning.
Scenario B: Build a dedicated singing voice cloning model from scratch
A from-scratch singing model that works across arbitrary voices needs significantly more data:
- Recommended: 200 to 500 hours of clean, diverse singing data from at least 50 unique vocalists.
- Minimum viable: 50 to 100 hours from at least 15 unique vocalists for a narrower scope.
- Per-voice zero-shot cloning: Possible with sufficient base data, using a 6 to 30 second reference clip at inference time.
Scenario C: Enable users to clone their own voice for singing
This is the most common consumer feature request. Users upload a short sample of themselves singing and get a model that can generate new singing in their voice. The base model handles everything except the user's identity; the user-uploaded data only needs to capture voice characteristics.
- User-side requirement: 5 to 15 minutes of the user singing, ideally in a range and style similar to the target outputs.
- Platform-side requirement: A pre-trained singing base model built from 100+ hours of diverse licensed singing data.
The legal structure you need
Voice cloning has a particular legal exposure that general music AI does not: the outputs can produce recognizable copies of specific voices. Even if your training data is fully licensed, if the output is a recognizable clone of a named person's voice without their authorization, you may face right-of-publicity claims, BIPA claims (in Illinois), or Tennessee ELVIS Act claims depending on jurisdiction.
Licensed training data does not fully protect you against these claims, but it is a necessary precondition. The legal structure for a voice cloning product built on licensed data typically includes:
