AI stem splitters separate sounds but can't prevent bleed
Marta Ilves, Audio tester (AI-assisted) ยท
AI stem splitters work by converting audio into a time-frequency representation, then using neural networks to assign parts of the sound to stems such as vocals or drums. However, when these sounds overlap in the spectrogram, energy from one source—like a voice—can spill into another, such as the drums. This spill, or "bleed," is a limitation of how the underlying transforms and models handle real-world music.
AI stem splitters: what they actually do
AI stem splitters start by transforming an audio file into a spectrogram using the short-time Fourier transform (STFT) [1]. This process breaks the audio into small, overlapping chunks (windows), and for each chunk, it calculates which frequencies are present at each moment. The result is a two-dimensional map: time on one axis, frequency on the other, with color or intensity showing the energy at each point.
Once the audio is in this format, a deep learning model takes the spectrogram and tries to allocate each piece—each time-frequency bin—to a particular source: vocal, drums, bass, or other [3]. The model uses patterns it has learned from training data to make these assignments.
If all the sounds in a song occupied separate, non-overlapping frequency ranges, this would be straightforward. But in real music, vocals and drums often share frequencies, and both may be present at the same moment. The splitter must decide, for each region, which stem it belongs to. Where the model is uncertain or the energy is mixed, errors and bleed occur.
The role of the window: why spectral leakage happens
The STFT uses a window function to focus on a short slice of audio at a time [1]. This window is finite in length. Mathematically, this is necessary, but it comes with a side effect: spectral leakage [2]. When you analyze only a part of a signal, energy from one frequency can "leak" into adjacent frequencies in the resulting spectrogram.
This leakage means that even if a vocal is singing at a frequency close to a snare drum hit, its energy will not be confined to a single bin. Instead, it will spill over, blurring the boundary between sources. Every window function (Hann, Hamming, rectangular, etc.) has its own tradeoff between frequency resolution and leakage [2].
For real music, this means that no matter how sharp the model, the data it starts from is already smeared by the transform. As a result, a vocal’s energy may appear in the bins the model wants to give to the drums, and vice versa.
Deep learning’s approach: strengths and limits
Deep learning models used for stem separation do not operate directly on raw audio—they rely on spectrograms, typically produced by the Short-Time Fourier Transform (STFT) [3]. These models learn from examples to associate patterns in the spectrogram with different sources, such as vocals or drums.
Neural networks can extract cues like vocal contours or drum transients from the spectrogram, but their output is fundamentally limited by the resolution and overlap in the input representation. When the STFT merges the energy from overlapping sources, the neural network cannot fully disentangle what was mixed together in the time-frequency domain—no matter how sophisticated its architecture or how extensive its training data [3].
No source in the material quantifies the amount of bleed between stems; what the sources do show is that deep learning models are constrained by the information available in the spectrogram, and cannot recover details lost or blurred by the STFT step [3]. So, for any track where sources overlap significantly in the same frequency bands at the same moment, the model’s best output will still be a compromise, shaped by these technical limits.
A worked example: why vocals bleed into the drums
This is a hypothetical illustration, not a measured result. Suppose a track where a singer sustains a note as a snare hits. With a Short-Time Fourier Transform (STFT) using a 1024-sample window (about 23 ms at 44.1 kHz), both events overlap—vocals and snare share time and some frequency space. Because the window is finite, the vocal's energy is not perfectly confined; instead, it spreads into neighboring frequency bins, including those occupied by the snare [2]. When a neural network tries to allocate this region, it faces a mixture: the sharp transient of the snare blurred with the vocal’s sustained energy. If the model assigns this segment to the drums, some vocal content leaks into the drum stem; if it assigns it to vocals, percussive energy may appear in the vocal stem. Spectral leakage from windowing ensures some cross-contamination: the separation can’t be perfect [2].
What this means for your workflow
If you use AI stem splitters for remixing, karaoke, or mastering, you will notice that extracted stems are never perfectly isolated. Vocal bleed into percussion is most obvious during loud, sustained notes that coincide with drum hits. The underlying reason is not an error in the AI model but a physical limit of the time-frequency transform and spectral leakage [2].
You can experiment with different window sizes or models, but the tradeoff remains: shorter windows give better time resolution (sharp attacks) but more leakage, while longer windows give better frequency resolution but blur attacks. No setting removes bleed when sources overlap in both axes [1,2].
Questions readers ask
Can I change the stem splitter’s settings to reduce vocal bleed?
You can try adjusting the window size or using a model trained for a specific genre, but these changes only shift the tradeoff between time and frequency resolution. They cannot eliminate bleed where sources overlap.
Will future AI models solve the bleed problem?
Better models may reduce artifacts and make smarter guesses, but as long as they rely on STFT-based spectrograms, the mathematical limitation remains. Only if a fundamentally different representation is used could perfect separation be possible.
Why don’t I hear the same amount of bleed in every song?
Bleed is worst when vocals and drums overlap in both time and frequency, and when one is much louder than the other. Sparse arrangements or stems with non-overlapping frequency ranges will show less bleed.