MultipleNegativesRankingLoss issue

[yfdjchvlgv]

When I want to use MultipleNegativesRankingLoss, my dataset consists of a query with one positive example and multiple negative examples, but the number of negative examples can range from 0 to 5. When I try to format it as (anchor, positive, negative_1, …, negative_n), it seems necessary to structure it as (anchor, positive, [negative_1, …, negative_n]). Additionally, the list of negative examples must contain more than two negative examples for proper training. The same dataset can train an embedding model, but when training a reranking model, it throws an IndexError: The shape of the mask [2, 4] at index 0 does not match the shape of the indexed tensor [4, 2] at index 0.

[yfdjchvlgv]

The error occurs after completing a training epoch and finishing the computation of the validation set loss.
100%|██████████| 52/52 [00:18<00:00, 6.49it/s]
53it [00:18, 6.90it/s]
54it [00:18, 7.03it/s]Traceback (most recent call last):
File "\lib\environment\lib\site-packages\transformers\trainer.py", line 2325, in train
return inner_training_loop(
File "\lib\site-packages\transformers\trainer.py", line 2790, in _inner_training_loop
self._maybe_log_save_evaluate(
File "\lib\site-packages\transformers\trainer.py", line 3221, in _maybe_log_save_evaluate
metrics = self._evaluate(trial, ignore_keys_for_eval)
File "\lib\site-packages\transformers\trainer.py", line 3170, in _evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "\lib\site-packages\sentence_transformers\trainer.py", line 543, in evaluate
return super().evaluate(eval_dataset, ignore_keys, metric_key_prefix)
File "\lib\site-packages\transformers\trainer.py", line 4489, in evaluate
output = eval_loop(
File "\lib\site-packages\sentence_transformers\trainer.py", line 553, in evaluation_loop
output = super().evaluation_loop(
File "\lib\site-packages\transformers\trainer.py", line 4685, in evaluation_loop
losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
File "\lib\site-packages\transformers\trainer.py", line 4902, in prediction_step
loss, outputs = self.compute_loss(
File "\lib\site-packages\sentence_transformers\trainer.py", line 431, in compute_loss
loss = loss_fn(features, labels)
File "\lib\site-packages\torch\nn\modules\module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "\lib\site-packages\torch\nn\modules\module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "\lib\site-packages\sentence_transformers\cross_encoder\losses\CachedMultipleNegativesRankingLoss.py", line 247, in forward
for negatives in self.get_in_batch_negatives(inputs[0], inputs[1:]):
File "\lib\site-packages\sentence_transformers\cross_encoder\losses\MultipleNegativesRankingLoss.py", line 148, in get_in_batch_negatives
negative_indices = all_indices[mask].reshape(batch_size, -1)
IndexError: The shape of the mask [2, 4] at index 0 does not match the shape of the indexed tensor [4, 2] at index 0

[tomaarsen]

Hello!

The most straightforward format will be:

(anchor, positive, negative_1)
(anchor, positive, negative_2)
(anchor, positive, negative_3)
...

But if you want to use (anchor, positive, [negative_1, …, negative_n]) to train a CrossEncoder with MultipleNegativesRankingLoss, you'll likely have to subclass the loss and update get_in_batch_negatives so it recognizes the different format for the negative column. You'll likely also have to update call_model_with_columns so that it works with candidates being a nested list.
In short: the format of (anchor, positive, [negative_1, …, negative_n]) isn't supported by the loss out of the box, although it can be added.

There's other losses that do support roughly this format, the learning-to-rank style losses, e.g. LambdaLoss. See the Loss Overview, it shows that you can process (anchor, [positive, negative_1, ..., negative_n]) with a scores or labels column like [1, 0, ..., 0]. See for example https://github.com/huggingface/sentence-transformers/blob/main/examples/cross_encoder/training/ms_marco/training_ms_marco_lambda.py.

I believe that it's only possible to use lists of strings as as training dataset column with CrossEncoder models, the SentenceTransformer class won't be able to tokenize them correctly.

• Tom Aarsen

[yfdjchvlgv]

However, based on my practical experience, fine-tuning an embedding model using data in the format (anchor, positive, [negative_1, …, negative_n]) is feasible. However, when training a reranking model, it throws an error like this.

If we uniformly change it to

(anchor, positive, negative_1)

(anchor, positive, negative_2)

(anchor, positive, negative_3)

and so on, compared to the format I intended to use (anchor, positive, [negative_1, …, negative_n]), would this increase the weight of the positive sample because it is repeated?

[yfdjchvlgv]

What's more troublesome is, if I have multiple positive and negative samples for a single anchor, how should I handle it? Would decomposing them into the format (anchor, positive, negative) result in too many data points?

[yfdjchvlgv]

In other words, if I want to use data in the format (anchor, positive, negative_1, …, negative_n) for training, is it necessary to ensure the number of negative examples is the same?

[tomaarsen]

However, based on my practical experience, fine-tuning an embedding model using data in the format (anchor, positive, [negative_1, …, negative_n]) is feasible.

Hmm, when I tried it myself earlier, it didn't work:

from datasets import load_dataset
from torch import Tensor
import random
from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    SentenceTransformerModelCardData,
)
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers


# 1. Load a model to finetune with 2. (Optional) model card data
model = SentenceTransformer(
    "microsoft/mpnet-base",
    model_card_data=SentenceTransformerModelCardData(
        language="en",
        license="apache-2.0",
        model_name="MPNet base trained on AllNLI triplets",
    ),
)

# 3. Load a dataset to finetune on
dataset = load_dataset("tomaarsen/gooaq-hard-negatives", "triplet-5", split="train")


def map_to_varying_negatives(example):
    negatives = [example[f"negative_{i}"] for i in range(1, 6)]
    negatives = random.sample(negatives, k=random.randint(1, 5))
    return {"question": example["question"], "answer": example["answer"], "negatives": negatives}


dataset = dataset.select(range(100_000))
dataset = dataset.map(map_to_varying_negatives, batched=False, remove_columns=dataset.column_names)
train_dataset = dataset
print(train_dataset)
print(train_dataset[0])
print(train_dataset[1])

# 4. Define a loss function
loss = CachedMultipleNegativesRankingLoss(model)

# 5. (Optional) Specify training arguments
run_name = "mpnet-base-gooaq-varying-negatives"
args = SentenceTransformerTrainingArguments(
    # Required parameter:
    output_dir=f"models/{run_name}",
    # Optional training parameters:
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,  # Set to False if you get an error that your GPU can't run on FP16
    bf16=False,  # Set to True if you have a GPU that supports BF16
    batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
    # Optional tracking/debugging parameters:
    save_strategy="steps",
    save_steps=0.1,
    save_total_limit=2,
    logging_steps=0.01,
    run_name=run_name,  # Will be used in W&B if `wandb` is installed
)

# 6. (Optional) Create an evaluator & evaluate the base model

# 7. Create a trainer & train
trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    loss=loss,
)
trainer.train()

# (Optional) Evaluate the trained model on the test set

# 8. Save the trained model
model.save_pretrained(f"models/{run_name}/final")

# 9. (Optional) Push it to the Hugging Face Hub
model.push_to_hub(run_name)

Traceback (most recent call last):
  File "c:\code\sentence-transformers\demo_3581.py", line 75, in <module>
    trainer.train()
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\trainer.py", line 2325, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\trainer.py", line 2618, in _inner_training_loop
    batch_samples, num_items_in_batch = self.get_batch_samples(epoch_iterator, num_batches, args.device)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\trainer.py", line 5654, in get_batch_samples
    batch_samples.append(next(epoch_iterator))
                         ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\accelerate\data_loader.py", line 566, in __iter__
    current_batch = next(dataloader_iter)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\torch\utils\data\dataloader.py", line 734, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\torch\utils\data\dataloader.py", line 790, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\torch\utils\data\_utils\fetch.py", line 55, in fetch
    return self.collate_fn(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\data_collator.py", line 116, in __call__
    tokenized = self.tokenize_fn(inputs, task=task)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\SentenceTransformer.py", line 1623, in tokenize
    return self[0].tokenize(texts)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\models\Transformer.py", line 309, in tokenize
    batch2.append(text_tuple[1])
                  ~~~~~~~~~~^^^
IndexError: list index out of range
  0%|          | 0/6250 [00:00<?, ?it/s]

And yes, it would increase the weight of the anchor-positive pair, which I recognize is not ideal.

What's more troublesome is, if I have multiple positive and negative samples for a single anchor, how should I handle it? Would decomposing them into the format (anchor, positive, negative) result in too many data points?

It wouldn't necessarily result in too many datapoints, but it is something to be mindful of, yeah. For rerankers, I would have probably still gone with a BinaryCrossEntropy style loss with pairs with 1 or 0 labels for positive and negative, respectively. BCE is surprisingly tricky to beat in my experience.

Is it necessary to ensure the number of negative examples is the same?

If you want to use the (Cached)MultipleNegativesRankingLoss for rerankers, then sadly yes.

• Tom Aarsen

[yfdjchvlgv]

Indeed, when fine-tuning the reranking model, I have used BinaryCrossEntropyLoss, LambdaLoss, and ListMLELoss. Comparatively, BinaryCrossEntropyLoss has proven to be the most effective. However, during the fine-tuning of the embedding model, MultipleNegativesRankingLoss yielded the best results, which is why I intend to apply this loss function to the reranking model fine-tuning as well. Below is the function I used to convert the dataset format:
Ensuring the negative example list has a minimum length of 2 is crucial, otherwise my code will also throw an IndexError: list index out of range.

def convert_to_dataset(data):
# Convert data format
transformed_data = {
"anchor": [],
"positive": [],
"negative": []
}

for item in data:
    anchor = item["query"]
    for pos in item["pos"]:
        transformed_data["anchor"].append(anchor)
        transformed_data["positive"].append(pos)
        # Handle empty or single negative cases
        if not item["neg"]:
            item["neg"].append("This is a default negative example")
            item["neg"].append("This is a default negative example")
        if len(item["neg"]) == 1:
            item["neg"].append("This is a default negative example")
        transformed_data["negative"].append(item["neg"])

return Dataset.from_dict(transformed_data)

[yfdjchvlgv]

I've identified the issue with feeding negative examples as a list input. When the input is a list and the number of list elements exceeds 2, the tokenize function in the sentence_transformers\models\Transformers.py file enters the else condition during the following conditional check. It assigns the first two elements of the list to batch1 and batch2, but it's unclear what subsequent processing is done.

Currently, with an variable number of negative examples, my approach is to set a fixed number of negative examples to use. If the available negative examples exceed this number, only that many will be selected. If there are fewer, random non-positive example segments will be drawn from the total text to supplement the count until the required number of negative examples is reached.

def tokenize(
    self, texts: list[str] | list[dict] | list[tuple[str, str]], padding: str | bool = True
) -> dict[str, torch.Tensor]:
    """Tokenizes a text and maps tokens to token-ids"""
    output = {}
    if isinstance(texts[0], str):
        to_tokenize = [texts]
    elif isinstance(texts[0], dict):
        to_tokenize = []
        output["text_keys"] = []
        for lookup in texts:
            text_key, text = next(iter(lookup.items()))
            to_tokenize.append(text)
            output["text_keys"].append(text_key)
        to_tokenize = [to_tokenize]
    else:
        batch1, batch2 = [], []
        for text_tuple in texts:
            batch1.append(text_tuple[0])
            batch2.append(text_tuple[1])
        to_tokenize = [batch1, batch2]