MUSICBRAINZ_TRACKID is not truly unique in the MB Database (the field mapped to that tag)

[mikeysas]

This underlying issue was discovered when @crystalgypsy reported on the forum that a newly added album was not showing on the New Music list and eventually determined the reason was that the album contained a track with the same MBID as an older compilation album in their library. You can pick the thread at my post here.

I was not aware, but apparently between 2011 - 2013 MB changed their database internally and now have a RecordingID (unique to a recording but will be duplicated when that recording appears on multiple releases) and TrackID (unique to a recording, release, and track# with no duplicates). But in an attempt to not break existing tagging in folks libraries, the decision was made to map tags as follows (confusing to me):

• MUSICBRAINZ_TRACKID = internal RecordingID (not unique)
• MUSICBRAINZ_RELEASETRACKID = internal TrackID (unique)

References confirming the above:
https://picard-docs.musicbrainz.org/en/appendices/tag_mapping.html
https://community.metabrainz.org/t/what-does-musicbrainz-trackid-signify/650381

**Do note to make this even more annoying to think about, where as Picard does import both of the tags above, the standard MusicBrainz import for mp3tag still only imports MUSICBRAINZ_TRACKID. You need to install the MusicBrainz Expanded webscript to import MUSICBRAINZ_RELEASETRACKID.

I am documenting this issue so it is not lost, but not sure what the best solution is here and it is likely worthy of more discussion. I am not sure enough folks who rely on MBID tagging saw the Forum post on New Music. I wanted to post here and get your opinion @michaelherger first before creating a forum thread specific to this (if that would even be helpful). I appreciate your insight.

Additional thoughts:

• Technically this has been an "issue" for a while, but I guess those that have this situation have not noticed that they had more than one track updating play stats and ratings on the same record in persist.db. Easy not to notice until someone had added a New album in 9.x with this situation.
• Although the simple looking solution could be using MUSICBRAINZ_RELEASETRACKID, I think we need to be cautious about how that would impact folks including those that have MUSICBRAINZ_TRACKID only (maybe that could still be a fallback).
• In the forum post @crytalgypsy proposed why not use the combination of Album and Track MBID's, but that may open other questions too.
• The general issue of a user relocating or reorganizing their music library is still a problem that comes up and the New Music logic has only highlighted that more. Is it worth exploring other alternatives for that underlying problem that the current MBID logic is intended to handle?
• I don't think this is an urgent problem that should be jumped on without being carefully thought about, but it is something in my opinion to be considered at some point as I don't think the current situation is ideal.

[mikeysas]

Good additional discussion on this in this thread and summarize on my post:
https://forums.lyrion.org/forum/user-forums/logitech-media-server/1763424-duplicate-files-with-identical-musicbrainz-track-ids?p=1763983#post1763983

I believe that right suggestion here to consider as minimum step is for the scanner to log warnings when it reads a duplicate mb_trackid on a track compared to library.db tracks table and add the track without the mbid. This will maintain the integrity of 1:1 match between tracks table in library.db and persist.db eliminating the anomalies that folks have encountered and should not adversely impact those who have not seen an issue.

Additionally or subsequently if there is interest in some of the other suggestions in that thread like considering an alternate unique id tag in addition to mb_traqckid that can certainly improve a user's experience, but IMHO, I see at least preventing duplicates to maintain the 1:1 integrity as the higher priority here and required regardless of any of the other options discussed so far.

[darrell-k]

Proposed change:

In brief (more thought required)

1. Add LMS_PERSISTENT_ID column to the tracks table.
2. Amend the scanner to write the LMS_PERSISTENT_ID tag to the new column if it exists.
3. Add code to the scanner to identify existing duplicate MUSICBRAINZ_ID values in the tracks table. Write informational messages to the scanner log and do not carry out the actions below until duplicates have been resolved.
4. Amend the scanner to populate the ID in the persistent tracks table for new tracks with LMS_PERSISTENT_ID or MUSICBRAINZ_RELEASETRACK_ID or MUSICBRAINZ_TRACK_ID in this order of preference. For existing tracks, update the ID in the same order of preference, but only if the existing ID matches one of the 3 possibilities. For example, if the persistent table contains the same MUSICBRAINZ_TRACK_ID as the scanned track, but the scanned track is also tagged with a higher priority tag, update the persistent ID to the higher priority tag.
5. Ensure the persistent track id is updated during a new and changed scan (at present it only occurs in a full rescan).

[michaelherger]

Gents - I adore your enthusiasm and motivation. But I'm still struggling to understand the issue. People are using MBID, but don't like them, therefore we shall introduce our own unique identifier? Why not get rid of MBID in the first place in those cases? Or replace them with what you'd create as the LMS ID? UUIDs are supposed to be unique over time and space. You can create your own MBID and should not have to fear any conflict.

My opinion is that we won't fix the fundamental problem by adding more complexity and obscure own tags. This might help the hand full of super vocal users discussing the topic. But it'll remain hidden, irrelevant and esoteric for 99.9 plus some percent of the users. And they'll have to deal with the bugs introduced or caused by the additional logic.

To quote one of those users: "all of the half-assed standards conjured in decades past are piss-poor". Isn't this proposed solution just adding another item to that collection?

Some of you or the participants in that discussion are using other software to manage their music collection. Is there anything to learn from them?

[michaelherger]

BTW: LMS_PERSISTENT_ID would be a custom tag the user would have to define themselves in the tagger of their choice?

[mikeysas]

Gents - I adore your enthusiasm and motivation. But I'm still struggling to understand the issue.

The issue is that the original implementation of MBID as an alternate link was done on the assumption that mb_trackid was unique, but MB subsequently changed that to no longer be true over 10 years ago and now we have a data integrity issue where multiple track records in library.db can be linked to a single persist.db record. I believe the intended design for LMS assumes this is always 1 to 1 link. However until recently the side effects of this issue were only noticeable to those paying attention closely.

Apparently this has been a known issue for a while and as is typical with this community there is a small group of users who have been working around this for a while by manually identifying duplicates and replacing the duplicates manually with user generated UUID’s. There is another group of users who have this issue but have never noticed the impact and of course all the users that either don’t use MBIDs or have no duplicates don’t have an issue.

The reason that more users are recognizing this issue is that our recent change to New Music makes this issue more visible when a user adds a album to their collection that contains a duplicate mb_trackid and is surprised by why the album is not showing on the New Music list and posts on the forum that the New Music list is broken.

As I stated above, I think preventing duplicates should be the top priority to maintain the 1:1 integrity between library.db and persist.db which is covered in #3 from Darrell.

I can’t speak to the potential broader impacts from the complexity of what Darrel has proposed. What I like about his proposal is that it sounds like it will very cleanly and transparently address the issue for users with existing duplicates if they have mb_release_trackid tags which are unique. It also offers the use of a new custom tag for those users that wish to abandon MBIDs and create their own UUIDs or even a hybrid of both.

Does that help you understand the issue?

[michaelherger]

Yes, thanks, I believe I understand the issue. But I have doubts that relying on our own custom tag will solve any real problem. Yes, it will solve it for some. But it'll be an extremely small group which a.) experiences the issue, b.) bothers about it, c.) knows about this workaround, d.) uses a file format supporting custom tags, e.) uses a tagger allowing them to create custom tags, and f.) is willing to go all the way to define that custom, proprietary value. And to cover the desire of that extremely small group we'll add complexity which someone will have to maintain going forward. And all users will have to suffer from potential new bugs introduced by the change, slower processing etc.

Maybe we should also look at the severity of the issue? What are the consequences so far? Wrong playback count numbers and incorrect "new music" entries? Is there more?

[mikeysas]

Along with this change, I will help create a guide for tagging your music to ensure LMS can track your music across disk migrations or reorganizations of the directory structure. The whole reason MBID was added originally was to provide that alternate link for persist.db, but has always been flawed not just because of the duplicate issue that arose later, but also because MB has always been incomplete. Just look at how many posts there have been complaining about this fact. I believe we will see more users (especially those with large libraries) take advantage of this once they see an easy guide on how to do it which also explains the value of making the effort.

And yes, the group of users for this will still be a smaller minority, but it will be much larger than the group commenting in that thread and I expect it will grow over time.

[mikeysas]

Maybe we should also look at the severity of the issue? What are the consequences so far? Wrong playback count numbers and incorrect "new music" entries? Is there more?

Some other minor behavior oddities with Ratings and the behavior of some of AF1’s plugins I have also seen as a result of the duplicates.