Duplicate points (texts) getting clustered into multiple clusters

[PalrechaSayyam]

Hi,

I am working on a dataset that has a huge chunk of duplicate/near duplicate data, i am using HDBSCAN for clustering but i am getting multiple clusters of the duplicate data.

Ideally this shouldn't have happened and the whole chunk should have been in a single cluster, also i understand that it might be an affect of using UMAP on the BERT embeddings, but dimensionality reduction is required and can't be bypassed.
Also the idea of deduplication by dropping duplicates, perform HDBSCAN and then remap them to their designated clusters does works to a good extent but i am bound to NOT drop any datapoints for clustering.

I also came up with a method to not drop and remap the duplicates to the right cluster, but this feels just like a workaround and not a real fix.

Is there a proper fix for this issue, or is it a bug that HDBSCAN has based on how its designed...?

Thanks for your time and efforts.
Regards,
Sayyam

[dookie23]

Hi Sayyam,

we have similar datasets (with many duplicates) and HDBSCAN works correctly in our case.

In the past we had the same problem and we detected that the problem was in the embeddings. More specifically by reducing them with UMAP due to their stochastic nature.
https://umap-learn.readthedocs.io/en/latest/reproducibility.html

I would recommend that you:

• Check that two similar documents (texts) have exactly the same embeddings (to rule out that the problem comes when generating embeddings)
• Set a random_state in UMAP (e.g. umap.UMAP(random_state=42).fit(data))

I hope these tips can help you.

Regards,