have trobule reproducing LLaMA-3-8B-it BT

[jijivski]

When reproducing LLaMA-3-8B-it BT in RLHF Workflow: From Reward Modeling to Online RLHF A Comprehensive Practical Alignment Recipe of Iterative Preference Learning.
It outputs the same score for chosen vs rejected, but models like gemma-2b-it and Qwen2.5-0.5B-Instruct are right.
Could you please provide some clues about the environment or training config?

#huggingface/trl#2265

command

cd RLHF-Reward-Modeling/ 
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch --main_process_port 29502 ./bradley-terry-rm/gemma_2B_rm_orig.py --deepspeed deepspeed_configs/deepspeed_2.json --model_name Meta-Llama-3-8B-Instruct --max_length 4096 --train_set_path hendrydong/preference_700K --per_device_train_batch_size 2 --per_device_eval_batch_size 2 --gradient_accumulation_steps 4 --learning_rate 2e-05 --weight_decay 0.0005  --output_path bt_models/Meta-Llama-3-8B-Instruct/

environment

CUDA 12.1 (can this be the problem?)
environment just as RLHF-Reward-Modeling/bradley-terry-rm/README.md

python=3.10.9
pip3 install torch==2.1.2 torchvision torchaudio
pip install flash-attn==2.6.3
pip install accelerate==0.33.0 # for gemma2 and llama3.1
pip install deepspeed==0.12.2
pip install transformers==4.43.4
pip install numpy==1.26.4

how I find rewards are all same

def compute_loss(...)
        print(loss,{"rewards_j": rewards_j, "rewards_k": rewards_k})

       grad_fn=<IndexBackward0>)}
tensor(0.6914, device='cuda:2', dtype=torch.bfloat16, grad_fn=<NegBackward0>) {'rewards_j': tensor([[6.2188],
        [6.2188]], device='cuda:2', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>), 'rewards_k': tensor([[6.2188],
        [6.2188]], device='cuda:2', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>)}
tensor(0.6914, device='cuda:2', dtype=torch.bfloat16, grad_fn=<NegBackward0>) {'rewards_j': tensor([[6.2500],
        [6.2500]], device='cuda:2', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>), 'rewards_k': tensor([[6.2500],
        [6.2500]], device='cuda:2', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>)}
tensor(0.6914, device='cuda:0', dtype=torch.bfloat16, grad_fn=<NegBackward0>) {'rewards_j': tensor([[6.2500],
        [6.2500]], device='cuda:0', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>), 'rewards_k': tensor([[6.2500],
        [6.2500]], device='cuda:0', dtype=torch.bfloat16,
       grad_fn=<IndexBackward0>)}

I'm new to this area, thank you for great paper and code !
& Appreciate any clues !

[WeiXiongUST]

Hi, can you try the script of llama3_8B_rm.py?

The major difference is that llama does not have a padding token and setting the padding token to be the eos token will lead to the issue you encounter.

[jijivski]

Sorry for the late reply. Thanks, this worked!
I referred to some docs, and found something useful(thanks again) but still kind of confused,

note that tokenizer.padding_side is by default right, so here if pad_token is not set, it will be token1 token2 token3 <eos> <eos> so output will be meaningless, but according to https://huggingface.co/docs/transformers/en/pad_truncation

The padding argument controls padding. It can be a boolean or a string:
True or 'longest': pad to the longest sequence in the batch (no padding is applied if you only provide a single sequence).

Based on this, in a batch, one row (in the case of a single sequence in the batch) should not be padded, and thus should have an output different from the others. Nevertheless, I can observe(mentioned before in print rewards_j) that all the outputs are all the same. I'm wondering if there's something I'm missing in the implementation or understanding of these concepts. Is there a particular setting or step I might have overlooked?

[Tsaibw]

Hi jijivski, may I ask if you were able to successfully reproduce the LLaMA-3-8B-it BT results?
My results are somewhat different from the paper, but the only change in my setup is the transformer version, and I ran it on 4 GPUs with --per_device_train_batch_size 2 and --gradient_accumulation_steps 64.

Chat: 97.35
Chat Hard: 56.8
Safety: 84.01
Reasoning: 73.3

Thanks

[jijivski]

Hi jijivski, may I ask if you were able to successfully reproduce the LLaMA-3-8B-it BT results? My results are somewhat different from the paper, but the only change in my setup is the model version, and I ran it on 4 GPUs with --per_device_train_batch_size 2 and --gradient_accumulation_steps 64.

Chat: 97.35 Chat Hard: 56.8 Safety: 84.01 Reasoning: 73.3

Thanks

Sorry, we opted for Gemma and didn't continue using LLaMA3.
Gemma has been more widely adopted in reward modeling tasks and shows better learning performance in our experiments.
Maybe you can try it, and wish you good luck!
:-）