Why deduplication is the hardest part of a music catalog

Crawl a few thousand music torrents and you will not find a few thousand songs. You will find the same songs over and over, spelled slightly differently every time, and the job of the index is to notice.

The same song, twenty ways

One track can appear as any of these, and every one is a different string:

01 - Highway to Hell.mp3
01. AC/DC - Highway To Hell.flac
AC-DC_-_Highway_to_Hell_(Remastered).mp3
ACDC - Highway to Hell [1979] (2003 Remaster).flac
Highway To Hell (feat. nobody) - AC⁄DC.m4a

A human sees one song. A naive database sees five, and the catalog inflates with noise.

Canonical keys

The fix is a canonical key: a hard normalised form of artist and title that survives punctuation, case, accents and separators. Lowercase it, strip diacritics, remove everything that is not a letter or a digit, then join the two fields.

"AC/DC"  + "Highway to Hell"   -> acdc|highwaytohell
"AC-DC"  + "Highway To Hell"   -> acdc|highwaytohell
"ACDC"   + "highway to hell"   -> acdc|highwaytohell

Three spellings, one key. Put a unique constraint on that key and duplicates cannot enter the catalog at all. Instead of a second row, a repeat becomes an extra source attached to the row that already exists.

Where it goes wrong

Normalise too aggressively and you merge songs that are genuinely different. These are traps worth respecting:

• Live and studio versions. The same artist and title, genuinely different recordings. Version needs to be its own field, not something you flatten away.
• Remixes and edits. A radio edit is not the album cut, and a broadcaster may care a great deal which one airs.
• Covers. Same title, different artist. The artist half of the key does the work here.
• Collaborations. "Herbie Hancock & Carlos Santana" is a joint credit, not a duplicated prefix to strip.

Scene names are not metadata

Release names are built for filesystems, not for reading:

Amy-Macdonald--Life-In-A-Beautiful-Light-(Deluxe)-(2012-Pop)-[Flac-16-44]

Hyphens stand in for spaces, the year and genre are welded on, and the format tag trails the end. Worse, some releases carry a site watermark as a file, so a naive crawler happily indexes www.some-torrent-site.com.mp3 as though it were a song. An index that does not filter those ends up with cement mixers and gas canisters filed under salsa.

None of this is glamorous. It is string handling, edge cases, and a long list of things that are almost the same. But it is the difference between five million songs and five million rows.