A hidden test set judged music demixing models in 2021

The tool moment

In 2021, Sony Group Corporation, with INRIA, organized the Music Demixing (MDX) Challenge, a competition asking researchers to separate stereo songs into four stems: vocals, drums, bass, and everything else. The challenge's own hosting page records that Round 1 ran from 3 May to 13 June 2021 and Round 2 from 14 June to 31 July 2021, with a final leaderboard published in early August. Entrants trained on the MUSDB18-HQ dataset or on data of their choosing, and every submission was scored against a hidden test set that Sony Music Entertainment (Japan) built and kept inaccessible to competitors. The organizing team, including Sony researcher Yuki Mitsufuji, later published a paper describing the challenge.

What the documents show

The paper states the challenge was designed to correct known limits in the field's standard MUSDB18 dataset, which it says reflects 'a focus on Western pop music and a limited number of mixing engineers.' The MDX Challenge instead drew on 'a wider range of music genres' and more mixing professionals building its confidential test set, and ranked entries using signal-to-distortion ratio, a numeric measure of how closely a separated stem matches a reference stem, averaged per song. The paper was first posted in August 2021 and later appeared in the peer-reviewed journal Frontiers in Signal Processing, so it has been through review, not just self-reported. Neither document names a winning score as representative of any commercial plugin; the results describe research systems evaluated on a dataset most listeners will never hear.

What stays with the musician

A leaderboard score from a research challenge says nothing about how a specific plugin performs on a specific song. A musician deciding whether to trust a product's separation quality cannot read across from an MDX ranking to that product unless the vendor states it uses a comparable model and reports its own score under the same method. Choosing which separation tool to use for a remix or a repair job remains a judgment the musician makes on the material in front of them, not on a competition result for different songs.

Judge it by listening

Signal-to-distortion ratio is a documented, numeric evaluation method, not a listening test, and the paper does not claim it substitutes for one. Editorially, a higher score on a hidden test set is evidence a research system separates that material more accurately by one defined measure; it is not evidence that a downstream product using a similar architecture will sound clean on a reader's own track, which still has to be checked by ear.

• Does a vendor claiming state-of-the-art separation cite a specific benchmark, dataset, and score, or only the phrase itself?
• Was the score produced on a hidden test set the model never saw, or on data it may have trained on?
• Does the measured metric match what I am listening for, such as bleed at transients, rather than an average distortion number?

The Music Demixing Challenge is strong evidence about how research models performed on one hidden dataset in 2021; it is not a verdict on any product a reader might buy today.