Is a Karaoke Backing Track Just the Original Song With the Voice Removed? — 50 Years of Source Separation, and One Song Split Seven Ways

Korean karaoke remotes have a "Melody" button. Press it and the tune of the song plays on top of the backing track. That always puzzled me. If the backing track was made by erasing the voice from the original recording, how could you switch just the melody back on?

The answer is anticlimactic. Karaoke backing tracks are not the original with the voice removed. They are built separately from the start, and the melody line is made at the same time, which is why it can be toggled.

But things are different now. Clips of K-pop music shows with the backing track stripped away, leaving only the singers' live voices, circulate online as "MR removed" videos (in Korea, a backing track is called an "MR"). With one free program you can pull the vocals out of almost any song. For a long time the common wisdom was that mixed sounds can't be un-mixed. When and how did that stop being true?

What I wanted to do

I wanted to map where the field of audio source separation started and how far it has come: the papers that led it, the public repositories, and the places it is actually used, laid out in chronological order.

This time it was hard to attach an interactive demo to the post. A tool that splits any uploaded song would turn the blog into a channel for uploading and downloading other people's recordings. So instead I split the same songs myself with methods from each era and included the scores and images as examples.

Karaoke tracks aren't erased — they're rebuilt

In a 2025 company post, TJ Media, Korea's largest karaoke company, explained how it makes backing tracks. Regular tracks are built with virtual instruments on a computer. For popular songs, session musicians play guitar, bass and saxophone live in the company's own studio. A live-played track takes more than a month and costs over five times as much as a regular one. The company produces around 300 songs a month this way.

Kumyoung, the other major Korean brand, works the same way. According to a 2009 Hankook Ilbo article, its production team picked out pitches and rhythms by ear, without sheet music, and mixed in sounds recorded by real session players and backing singers.

Japan is no different. Nobuhiro Kaneko, who has made backing tracks at the karaoke company JOYSOUND for more than 30 years, said in a 2025 interview that they still buy the CD, listen to each instrument one by one, and type them into a computer as numbers. A song takes three days at the fastest. Asked why they don't just extract the instruments from the CD, he replied:

Once fruit juice has gone through a blender, you can't separate it back into apples and oranges.

There is another reason. Record labels, as a rule, do not hand their original recording files to karaoke companies. Karaoke tracks have to be rebuilt both for technical reasons and for rights reasons.

A note on the term: "MR" is used only in Korea. Korean media expand it variously as Music Record, Music Recording or Music Recorded, and I couldn't confirm its exact origin. In English the usual terms are instrumental, backing track or karaoke version.

The first method: subtract the center

Attempts to un-blend the juice go back further than you might think. The oldest trick is simple. In a stereo recording the voice is usually placed dead center, meaning it is equally loud in the left and right speakers. Subtract the right signal from the left, and whatever sits in the center cancels out.

A patent Sony filed in Japan in 1973 (US Patent 3,941,931) describes a device that uses this principle to remove the centered voice and mix in a microphone in its place. For years afterwards, the "vocal remover" in karaoke machines and PC software mostly worked this way. The Korean "MR removed" videos that fueled debates about idols' singing ability around 2009 were, according to news reports at the time, made with "programs that can adjust the tone". The same articles noted that the backing track, backing vocals and crowd noise never disappeared completely.

The problem is that the voice is not the only thing in the center. Bass and kick drum are usually placed there too. MUSDB18, a research dataset, provides each song as separate vocals, drums, bass and other instruments. I applied left minus right to its 50 test songs and measured how much of each instrument survived.

While the voice dropped to 8%, the bass fell to 1% and the drums to 6%. That is why backing tracks made this way sound thin, missing their low-end thump. The reverb on the voice is spread across left and right, so it doesn't cancel and lingers like a ghost. And the result is mono, a drawback already noted in patent filings from the 1990s.

Splitting with math: from the cocktail party to matrix factorization

Even at a noisy party, you hear it when someone says your name. In 1953 the British scientist Colin Cherry studied this experimentally, and the task of picking out one sound from many came to be called the "cocktail party problem".

There were broadly two ways to solve it on a computer.

• Independent component analysis (ICA, 1995). Proposed by Bell and Sejnowski. When several microphones each pick up a different blend of sounds, you unmix them so the outputs are as unrelated to each other as possible, recovering the original sources. But you need as many microphones as sources, so it struggles with a two-channel stereo track containing ten or more instruments.
• Non-negative matrix factorization (NMF, 1999). Published in Nature by Daniel Lee and Sebastian Seung. It takes the time-frequency picture of a sound (a spectrogram) and splits it into "a handful of timbres" and "when each timbre plays". In 2003 Smaragdis and Brown used it to transcribe piano music, and in 2007 Virtanen refined it for music separation.

Around 2010, methods that exploit the nature of music appeared. REPET (Rafii and Pardo, 2012) uses the fact that accompaniment repeats bar after bar while the voice doesn't. RPCA (Huang et al., 2012) treats the accompaniment as low-rank and the voice as sparse. HPSS (FitzGerald, 2010) separates sharp, percussive bursts from sustained sounds.

In all of these, humans wrote the rules. They looked convincing on songs that fit the rules, and fell apart when the accompaniment changed every bar or the voice held long notes.

Once there was an answer key, deep learning moved in

What made the move to deep learning possible was less an algorithm than an answer key. To teach a machine how to split a song, you need both the mix and the original separate parts.

• In 2006 Vincent, Gribonval and Févotte defined SDR, the score for separation quality. It expresses, in decibels, how small the leftover error is relative to the true source. Every +3 dB halves the error energy
• At the end of 2017 MUSDB18 was released: 150 songs (100 for training, 50 for testing), each split into vocals, drums, bass and other. Since then almost every paper reports scores on the same 50 test songs

After that, things moved quickly.

• 2017, U-Net (Spotify and City, University of London). Applied a network designed to outline objects in images to spectrograms, trained on roughly 20,000 pairs of songs and their instrumental versions
• 2019, Open-Unmix (Inria and Sony). A reference model with code and trained weights released so anyone could reproduce it
• 2019, Spleeter (Deezer). A public tool that did 2-, 4- or 5-way separation with a single command, about 100 times faster than real time on one GPU. For most people, this was their first encounter with "AI vocal removal"
• 2019–2022, Demucs (Facebook/Meta). Started from working directly on waveforms instead of spectrograms. Hybrid Demucs, which combined both, won the 2021 Music Demixing Challenge run by Sony (MDX 2021). HT Demucs, which added a Transformer, followed in 2022
• 2023, BS-RoFormer (ByteDance). Cuts the spectrum into bands and runs a Transformer on each band. It won the 2023 Sound Demixing Challenge (SDX 2023), and models from this family are now the default in public separation tools
• 2025, SAM Audio (Meta). Lets you name the sound you want in text, like "just the guitar", or click on an object in a video, and extracts only that sound. A move beyond the fixed four-stem frame for music

These are the vocal scores on the MUSDB18 test set as reported in each paper. They rose from around 3 dB in 2018 to over 12 dB in 2023, roughly an eighth of the error energy. Red outlines mark models trained with songs beyond MUSDB18. The same architecture gains 0.5–1 dB with more songs, and BS-RoFormer, which grew the model and added songs at once, gained more than 2 dB. Much of the competition in this field is a competition over training data.

Public repositories you can try

I split the same songs seven ways

On a laptop with an RTX 3070, I split 7-second excerpts of the 50 MUSDB18 test songs using seven methods from different eras. Because the true parts are available, each result can be scored directly. I used whole-clip SDR, as in the 2021 challenge, and took the median over the 50 songs.

It helps to set a baseline first. If you do nothing and hand over the original song as the "backing track", it still scores 4.5 dB, because the accompaniment is louder than the voice in the original.

• Left minus right scores -0.5 dB on the backing track. That's 5 dB worse than doing nothing (4.5 dB). Cutting away the bass and drums cost more than removing the voice gained
• REPET, which splits by rule, scores 3.3 dB. It doesn't clear the baseline either. With 7-second excerpts, it also had little time to find repeating bars
• The jump to deep learning with Open-Unmix (2019) brings it to 11.4 dB. That's 7 dB above the baseline, a fifth of the error energy
• BS-RoFormer, the default in today's public tools, scores 12.3 dB on vocals and 16.3 dB on the backing track. Another 5 dB above the 2019 model

Two caveats. The fine-tuned HT Demucs came out below the base version (8.4 vs 8.8), but with 50 seven-second clips a difference this small can flip on a few songs; by the mean the order reverses (8.5 vs 8.4). And while the BS-RoFormer score (12.3) is close to the paper's (12.7), the weights I used were trained and released by the community, not the paper's authors, and their training data isn't disclosed. If test songs were in the training set, the score may be inflated.

Scores alone don't give much of a feel for it, so I laid the sound out as pictures. Time runs left to right, pitch bottom to top, and brighter means louder. The artist released a separate vocal track for this song, so there is something to compare against (ccMixter, Mr_Yesterday, "Sins of the Fathers", CC BY 3.0).

In the original mix (top), the wavy lines of the voice overlap the horizontal lines of the instruments. The published a cappella (second) and the vocals extracted by BS-RoFormer (bottom) match almost exactly, down to the silent first second, the shape of the waves and the gaps for breaths. In left minus right (third), most of the vocal waves are gone, but the low end below 128 Hz has dimmed along with them. In the BS-RoFormer backing track (fourth), the bass band at the bottom stays as bright as in the original, and only the wavy lines are gone.

Where it went wrong

The two newest models returned "silent" vocals. The first time I ran BS-RoFormer and Mel-RoFormer, every one of the 50 songs scored exactly 0 on vocals. The vocal files were empty. At first I assumed the 7-second clips were shorter than the window the model looks at in one go (about 8 seconds), so I padded them with silence to 20 seconds. No change. The 20-second example song came out empty too. In the end I bypassed the separation tool (audio-separator) and loaded the same weights directly with the model authors' original code, which fixed it. The clue that something was off was the backing-track score: 4.5 dB, the same as the do-nothing baseline. In other words, the model hadn't done anything.

One panel of the spectrogram came out pitch black. The image for the left-minus-right backing track was completely black. To draw it I averaged the left and right channels, but this method outputs L-R on the left and R-L on the right, so the two add up to exactly zero. The audio itself was fine; I redrew that panel from a single channel.

The published a cappella was not an answer key. The example song comes with a separately released vocal track, so I tried to score against it. Comparing BS-RoFormer's extracted vocals to the a cappella gave -6.8 dB. The pictures look almost identical, yet the score is terrible. The voice in the mix differs in level from the published a cappella and went through other processing, so they don't line up waveform for waveform. I used this song only for pictures and took the scores only from MUSDB18, where the true stems are certain.

Installation got stuck three times. One component the separation tool needs (diffq) failed to install because it looked for a C compiler, and without it the path for calling Demucs was blocked, so I called Demucs separately. While the model weights were downloading, I thought progress had stalled and cut it off myself, then downloaded again. And a script waiting for the job to finish counted itself as "a job still running", so I waited 30 minutes for nothing.

Where it's used now

K-pop "MR removed" videos. From around 2009, videos that stripped the backing track from music-show performances fueled debates about singers' live skills. Back then it was little more than center subtraction, so much of the backing and crowd noise remained. Today, free tools remove it far more cleanly. Still, audio mixed with crowd noise and broadcast compression scores well below the experiment below. In the other direction, the Mnet show "Eumdampaeseol" in 2014 compared broadcast audio with the original recordings to spot lip-syncing.

The Beatles' last song. "Now and Then", released in 2023, is based on a cassette John Lennon recorded at home on piano in the late 1970s. His voice and the piano were stuck together on one track, which is why the song couldn't be finished in the 1990s. While making Peter Jackson's documentary "Get Back" (2021), his team developed a technique (MAL) for pulling individual instruments and voices out of mono recordings, and the same team used it to separate Lennon's voice from the piano. The 2022 reissue of "Revolver" was also remixed after its four-track tapes were split apart with this technique.

Karaoke modes in streaming apps. When Apple launched Apple Music Sing in December 2022, it let users turn a song's vocals up or down. A footnote in the press release says it "does not fully remove vocals".

Music production tools. Since May 2024, Apple's Logic Pro has included a feature that splits any recording into drums, bass, vocals and other. DJ software djay (2020) and Serato (2022) can drop the vocals out in real time during a set. The practice app Moises passed 30 million registered users in 2023.

Film and dubbing. In 2024 AudioShake launched a service that separates film audio into dialogue, music and effects. When an old film whose original stems are lost is dubbed into another language, it removes just the dialogue and keeps the music and effects.

And karaoke. Gaudio Lab, a Korean audio company, said in a 2024 interview that it had trialed its separation technology with a major Japanese karaoke company. It said about 90% of songs come out usable, while 10% fall short of commercial quality. Un-blending the juice is starting to become a realistic option for the karaoke industry too.

Being able to split it doesn't mean you can post it

As the technology gets easier, there are places to be careful.

• A backing track or vocal extracted from an original song is the sound of that recording itself. Beyond the songwriters' rights, the rights of the singers and musicians (performers) and of the label that made the recording (phonogram producer) still apply (Articles 69, 78 and 81 of the Korean Copyright Act). Using it alone at home to practice and posting it online are different matters
• Karaoke backing tracks are protected in their own right. TJ Media sued a YouTuber who used its karaoke machine without permission to make videos, and won at both the first and second instance; the appeals court awarded about 48 million won. The argument that "a backing track is a mere copy of the original and therefore not a phonogram" was rejected (on appeal to the Supreme Court as of March 2026)
• YouTube excludes covers that use commercial backing tracks from revenue sharing. Its help center says covers that use a commercial sound recording "such as an instrumental, karaoke recording" are not eligible for monetization. I couldn't confirm in official documentation whether a backing track extracted from the original gets matched by Content ID. But since it contains the original recording's accompaniment as is, it is reasonable to assume it likely will

Why one request is enough

AI invented almost nothing new here. The public dataset (MUSDB18), public models (Demucs, the RoFormer family) and public tools (audio-separator, the model-training repository) already existed; the AI's job was to install them, wire them together and score them. Installation got stuck three times, and the two newest models produced nonsense and had to be rerun. Leaving out the research, the experiments and the five figures took a little over two hours.

The real reason this field is "one click" is the answer key from earlier. A problem with a right answer can be compared by score, and a problem that can be compared by score gets pushed to its limit by ten years of papers and challenges. The results are published in a repository, so all that's left is installing it.

If you want to try it

• Decide what it's for first. For a practice backing track, pick a model that scores high on the accompaniment; to listen to the voice alone, pick a vocal-specific model. There are also dedicated "karaoke" models that remove only the lead vocal and keep the backing vocals
• Measure on songs with an answer key first. Run it on songs whose artists released an a cappella or instrument tracks (on ccMixter, for example), so you can compare with numbers rather than your ears
• Measure the do-nothing score first. If the result scores the same as just handing over the original song, the model didn't actually do anything. That comparison is what caught the silent output this time
• Run the old method alongside. Putting the left-minus-right result next to it makes it audible what the new models actually solved

What it can't do

• It doesn't separate perfectly. Even the best models let breaths and consonants bleed into the backing track when the singer belts. When an instrument doubles the vocal line, they confuse the two
• Bad recordings drop the scores sharply. These scores are for clean studio audio. Concert crowds, broadcast compression and phone fancams score much lower
• Backing vocals and harmonies are ambiguous. Most models lump every human voice into "vocals". If you want to keep the backing vocals the way karaoke does, you need a dedicated model
• It doesn't solve the rights. Being able to split a recording technically doesn't mean you can use someone else's recording

Karaoke companies still transcribe songs by ear less for technical reasons than for rights and quality. Even so, the juice can now be un-blended. Next time you press the "Melody" button on a karaoke remote, remember that someone wrote that tune down note by note.

If you enjoyed this, please leave a comment.

Sources

• Cover photo: Kumyoung karaoke remote, Catherine (Flickr), CC BY 2.0 (Wikimedia Commons.jpg))
• Example song for the spectrograms: "Sins of the Fathers" by Mr_Yesterday (ccMixter), CC BY 3.0. 20-second excerpt, used for the separation experiment
• How karaoke tracks are made: TJ Media, "This is how TJ karaoke music is made" (2025, Korean), Hankook Ilbo on Kumyoung's production team (2009, Korean), ORICON NEWS interview with JOYSOUND (2025, Japanese)
• "MR removed" videos: Maeil Shinmun (2009-02-20), Jaekyung Ilbo (2009-02-18), TenAsia on "Eumdampaeseol" (2014)
• Center-subtraction patents: US 3,941,931 (Sony, filed 1973); mono output drawback in US 5,550,920 (Mitsubishi)
• History: Cherry (1953) JASA · Bell & Sejnowski (1995) ICA · Lee & Seung (1999) NMF · Smaragdis & Brown (2003) link · Virtanen (2007) link · FitzGerald (2010) HPSS · Rafii & Pardo (2013) REPET · Huang et al. (2012) RPCA
• Evaluation and data: Vincent et al. (2006) BSS Eval · MUSDB18 · SiSEC 2018 · MDX 2021 · SDX 2023
• Deep learning: U-Net (Jansson et al. 2017) · Wave-U-Net · Open-Unmix · Spleeter · Demucs v2 · Hybrid Demucs · KUIELab-MDX-Net · BSRNN · HT Demucs · BS-RoFormer · Mel-RoFormer · SCNet · SAM Audio
• Scores in the by-year chart: Wave-U-Net, Open-Unmix, Spleeter and Demucs v2 from Table 1 of the Demucs v2 paper; Hybrid Demucs from Table 3 of its paper; KUIELab, BSRNN and HT Demucs from Table 1 of the HT Demucs paper; BS-RoFormer from Table 2 of its paper; Mel-RoFormer from Table 1 of its paper; SCNet from Table 2 of its paper. Values up to 2019 are on MUSDB18, later ones on MUSDB18-HQ (uncompressed versions of the same songs)
• Use cases: The Beatles, "Now and Then" announcement · "Revolver" special editions · Apple Music Sing · Logic Pro Stem Splitter · djay Neural Mix · Serato Stems · Moises user count (MBW) · AudioShake dialogue/music/effects separation · Gaudio Lab interview (Byline Network, Korean)
• Copyright: Korean Copyright Act Art. 69 · Art. 78 · Art. 81 · Art. 30, TJ Media backing-track lawsuit, appeals ruling (Herald Economy, 2026-03-04, Korean), YouTube cover monetization policy
• All experimental scores and figures in this post were computed by me. MUSDB18 is an academic-use dataset, so no audio or per-song figures from it are included