[SFT] Can't apply_chat_template prompt-completion dataset. so, training result always bad.

[HERIUN]

Reproduction

from datasets import load_dataset
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from trl import (
    SFTConfig,
    SFTTrainer
)

model_name = "Qwen/Qwen2.5-3B-Instruct"
dataset_name = "nvidia/OpenMathInstruct-2"
train_split = "train_1M"

raw_datasets = load_dataset(dataset_name, split=train_split)
dataset = raw_datasets.rename_columns({
        "problem" : "prompt",
        "generated_solution" : "completion"
    })

train_eval_dataset = dataset[train_split].train_test_split(test_size=0.1, seed=42)
train_dataset = train_eval_dataset["train"].take(1000)
eval_dataset = train_eval_dataset["test"].take(100)

training_args = SFTConfig(
    output_dir="/tmp",
)

model = AutoModelForCausalLM.from_pretrained(model_name)

trainer = SFTTrainer(
        model,
        args=training_args,  ## some training_args
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
    )

trainer.train()

In SFTTrainer'_prepare_dataset() doesn't apply chat_template to prompt-completion dataset. just prompt+completion+EOS

This works, but it doesn't seem to learn anything.(In my case repetition answer happen)

System Info

• Platform: Linux-6.11.0-1013-gcp-x86_64-with-glibc2.39
• Python version: 3.12.9
• TRL version: 0.17.0
• PyTorch version: 2.6.0
• CUDA device(s): NVIDIA L4, NVIDIA L4, NVIDIA L4, NVIDIA L4
• Transformers version: 4.51.3
• Accelerate version: 1.3.0
• Accelerate config: not found
• Datasets version: 3.5.0
• HF Hub version: 0.30.2
• bitsandbytes version: 0.45.5
• DeepSpeed version: 0.16.7
• Diffusers version: 0.33.1
• Liger-Kernel version: 0.5.9
• LLM-Blender version: 0.0.2
• OpenAI version: 1.76.0
• PEFT version: 0.15.2
• vLLM version: 0.8.5.post1

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete

[qgallouedec]

In SFTTrainer'_prepare_dataset() doesn't apply chat_template to prompt-completion dataset. just prompt+completion+EOS

Can you please precise what you mean by that?

[HERIUN]

@qgallouedec
In SFTTrainer._prepare_dataset()

first_example = next(iter(dataset))
if is_conversational(first_example):
    if isinstance(dataset, Dataset):  # `IterableDataset.map` does not support `desc`
        map_kwargs["desc"] = f"Applying chat template to {dataset_name} dataset"
    column_names = first_example.keys()
    dataset = dataset.map(
        apply_chat_template,
        fn_kwargs={"tokenizer": processing_class},
        remove_columns="messages" if "messages" in column_names else None,  # renamed to "text"
        **map_kwargs,
     )

It apply chat_template on messages(conversational) dataset. and I know some models include EOS token in their chat_template. ex. phi-4-mini(*does not Qwen2.5)

By the way, case in Not conversational data(just text or prompt-completion)

else:
    if isinstance(dataset, Dataset):  # `IterableDataset.map` does not support `desc`
        map_kwargs["desc"] = f"Adding EOS to {dataset_name} dataset"

    def add_eos(example, eos_token):
        if "text" in example and not example["text"].endswith(eos_token):  # language modeling case
            example["text"] = example["text"] + eos_token
        elif "completion" in example and not example["completion"].endswith(eos_token):
            example["completion"] = example["completion"] + eos_token
        return example

    dataset = dataset.map(
        add_eos,
        fn_kwargs={"eos_token": processing_class.eos_token},
        remove_columns="messages" if "messages" in column_names else None,  # renamed to "text"
        **map_kwargs,
    )
    # Subsequent tokenization will add special tokens (mostly for bos).
    # See https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens
    add_special_tokens = True

It just add EOS token at last.
But I think prompt-completion dataset also need chat_template like below text

In my case, finetuning Qwen2.5-3B-Instruct model with applied chat_template in prompt-completion data results good.
but, not apply chat_template(now SFTTrainer do) its results bad

[qgallouedec]

If I understand your point correctly, you're saying that even for plain-text datasets (what the docs call "Standard"), applying a chat template is important when fine-tuning instruction-following models?

The thing is, that doesn't quite make sense. You can only apply a chat template to conversational data, ie data that includes structured turns, like user/assistant messages.

But I think prompt-completion dataset also need chat_template like below text

depends if the data is conversational or not. Please refer to this section in the doc https://huggingface.co/docs/trl/en/dataset_formats#prompt-completion

[HERIUN]

I think prompt-completion(Standard format) also need applying chat template.

[qgallouedec]

I think there's a misunderstanding about the data format. Applying the chat model requires a specific data structure (containing the role etc).
If you try to apply the chat model to a standard format yourself, you'll see that it will fail.

[HERIUN]

But Now, trl dosn't support apply only_completion_loss on prompt-completion data(Conversational) like https://huggingface.co/datasets/trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness

The reason I created this issue was because it couldn't be done.

Any plan to support this?

I'm doing this by
standard prompt-completion -> conversational prompt-completion -> apply_chat_template -> standard prompt-completion data(kind of mess).

And it seems learn better than standard prompt-completion data with only_completion_loss
for qwen2.5-3B-instruct model

def extract_text_blocks(text, start_tag, end_tag_marker):
    blocks = []
    current_pos = 0
    while True:
        start_idx = text.find(start_tag, current_pos)
        if start_idx == -1:
            break

        search_for_end_tag_from = start_idx + len(start_tag)
        end_idx_of_marker = text.find(end_tag_marker, search_for_end_tag_from)
        
        if end_idx_of_marker == -1:
            break 

        block_end_pos = end_idx_of_marker + len(end_tag_marker)
        
        if text.startswith("\n", block_end_pos):
            block_end_pos += 1

        blocks.append(text[start_idx:block_end_pos])
        
        current_pos = block_end_pos
    
    return blocks


def convert_dataset(dataset, tokenizer, only_completion_loss):
    """
        prompt-completion format -> conversational format(messages) -> prompt-completion format
    """
    def process(example, tokenizer):
        msg = []
        msg.append({"role": "user", "content": example["prompt"]})
        msg.append({"role": "assistant", "content": example["completion"]})
        
        text = tokenizer.apply_chat_template(msg, tokenize=False)

        if tokenizer.bos_token is None:
            bos_token = tokenizer.special_tokens_map_extended["additional_special_tokens"][0]
            tokenizer.bos_token = bos_token
       
        splits = extract_text_blocks(text, tokenizer.bos_token, tokenizer.eos_token)
        if len(splits) != 3:
            logger.warning(f"While applying chat template, we assume 3 parts(system, user, assistant)"+
                           f"but its part has {len(splits)}")
        example["prompt"] = ''.join(splits[:-1])
        # prompt-completion dataset add eos later in SFTTrainer.init()
        example["completion"] = splits[-1].replace(tokenizer.eos_token, "")

        return example

    def convert_prompt_completion_to_messages(example):
        msg = []
        msg.append({"role": "user", "content": example["prompt"]})
        msg.append({"role": "assistant", "content": example["completion"]})

        return { "messages" : msg}

    if only_completion_loss:
        dataset = dataset.map(
            process,
            fn_kwargs={"tokenizer": tokenizer},
        )
    else:
        dataset = dataset.map(
            convert_prompt_completion_to_messages,
            remove_columns=dataset.column_names,
        )

    return dataset

train_dataset = convert_dataset(train_dataset, tokenizer, only_completion_loss)


trainer = SFTTrainer(
    model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset if training_args.eval_strategy != "no" else None,
    peft_config=peft_config,
)

[HERIUN]

I tested also phi-4-mini with/without chat_template. but it's ok. may only qwen2.5's problem