"How much data do I need?" is the first question every ML team asks when starting a voice model project, and it is the question with the least satisfying answer. The real answer is "it depends," and the dependencies matter more than any single number. This post breaks down the actual training data requirements across the current landscape of singing voice models, from two-minute fine-tunes on top of pretrained bases up to multi-hundred-thousand-hour frontier systems.
All numbers below come from published papers, GitHub documentation, or reported commercial specifications. Where a source is uncertain or contested, it is flagged.
The two modes: training from scratch vs fine-tuning
Before looking at specific numbers, understand the fundamental split. There are two training modes, and they have completely different data requirements.
Training from scratch means building a model without any pretrained starting point. The model learns everything — phoneme articulation, F0 control, timbre, musical phrasing — from your data alone. This is what research labs do when publishing new SVS architectures. It requires large datasets (hours to thousands of hours).
Fine-tuning means starting from a pretrained base model and adapting it to a specific task or voice. The base model already knows how voices work. You are only teaching it the specific characteristics of your target. Fine-tuning requires far less data (minutes to a few hours) because you are only adjusting the weights, not learning the fundamentals.
The number you need depends on which mode you are in. Most commercial teams do fine-tuning, not from-scratch training.
Fine-tuning a single voice: minutes to an hour
If your goal is to clone or adapt a specific voice on top of a pretrained base model, you need remarkably little data. The benchmark here is RVC (Retrieval-based Voice Conversion), which is the most widely used voice cloning tool in the open-source community.
RVC training requirements
The RVC project documentation states that "you can easily train a good VC model with voice data <= 10 mins." That is the lower bound. In practice, the quality curve looks roughly like this:
- 5 to 10 minutes: Minimum viable. Produces a usable model for simple conversions but with noticeable artifacts and limited generalization.
- 30 minutes to 1 hour: Recommended for good quality. Handles most phonetic contexts, pitch ranges, and dynamic variations reasonably well.
- 2 to 5 hours: High quality. Produces outputs that are difficult to distinguish from the target voice in most contexts.
- Beyond 5 hours: Diminishing returns for single-speaker fine-tuning. At this point the base model is the bottleneck, not the fine-tuning data.
The critical caveat is that the minutes have to be clean. 10 minutes of clean studio audio will outperform 60 minutes of noisy, bleed-contaminated audio. The RVC community has consistent reports that quality scales with data cleanliness faster than it scales with data quantity past the first 30 minutes.
So-VITS-SVC training requirements
So-VITS-SVC is another widely used voice conversion and cloning framework, oriented more toward singing than RVC. The documented minimums are slightly higher:
- 2 minutes minimum to produce a weak but functional model
- 1 to 2 hours recommended for decent quality per speaker
- 30+ hours recommended if training a new base model from scratch
So-VITS-SVC is more demanding on data quality than RVC because it is trying to preserve both the identity of the voice and the singing characteristics (pitch, vibrato, technique). The community recommendation is to prepare clean 5-to-15-second clips and ensure single-speaker isolation (no background vocals, no bleed).
XTTS v2 and other zero-shot cloning systems
Some modern systems claim to do zero-shot voice cloning from a single 6-second reference clip. XTTS v2 (Coqui, released 2023) is the most well-known example. The mechanism is different: the model does not update its weights, it encodes the reference clip into a speaker embedding and conditions generation on that embedding.
Zero-shot cloning is effectively "fine-tuning with zero data," because there is no gradient update. It works surprisingly well for TTS but is weaker for singing because the reference clip rarely captures the full vocal range and technique variations of the target singer. For production singing applications, teams still typically prefer a proper fine-tune on 10 to 60 minutes of clean data.
Training a new SVS model from scratch: hours
If you are training a new singing voice synthesis architecture and do not have a suitable pretrained base, the data requirements jump significantly.
DiffSinger
DiffSinger (Liu et al., AAAI 2022) is a widely cited reference architecture for singing voice synthesis. The original paper trained the acoustic model on PopCS, a Mandarin singing dataset of approximately 5 hours from a single female singer. The HiFi-GAN vocoder was trained on roughly 70 hours of singing data (though exact numbers in the secondary literature vary and should be verified against the GitHub repository).
Translated to practical terms: about 5 hours of clean single-speaker data is enough to train a publishable SVS acoustic model on a DiffSinger-style architecture. That is the floor for from-scratch research work. Production-quality systems typically want more.
VISinger and VISinger2
VISinger2 (2023) uses similar data scales to DiffSinger in its published benchmarks, typically training on OpenCpop (5.2 hours, single Mandarin singer). Multi-speaker variants train on OpenSinger (50 hours, 66 singers).
The pattern is consistent: 5-ish hours for single-speaker SVS, 50-ish hours for multi-speaker generalization. Below 5 hours, the model does not have enough phonetic coverage to synthesize arbitrary lyrics. Above 50 hours with diverse speakers, the model starts to generalize across voices and can be fine-tuned for new speakers with less data.
Multi-speaker scaling
The relationship between dataset size and model quality for multi-speaker SVS is not linear. Adding a new speaker improves generalization more than adding more data from an existing speaker, up to a point. Practical rules of thumb from the research literature:
- 1-5 speakers, 5-25 hours: Single-speaker or narrow multi-speaker model. Works for the trained speakers but does not generalize.
- 10-30 speakers, 30-100 hours: Moderate multi-speaker model. Generalizes to unseen speakers with fine-tuning on a small reference sample.
- 50-200 speakers, 100-500 hours: Strong multi-speaker model. Zero-shot cloning becomes possible for some speaker types.
- 200+ speakers, 500+ hours: Foundation-scale SVS. Approaches commercial production quality and supports downstream fine-tuning.
Frontier systems: tens of thousands to hundreds of thousands of hours
At the commercial frontier, the numbers get large fast.
