Iterative DPO

[Sanqiang]

I think authors of self rewarding llm didn't use standard DPO but Interactive DPO, which is from their another paper: https://arxiv.org/pdf/2312.16682.pdf.

[lucidrains]

is this the section you are looking at? they say it is similar to the cringe paper, but then goes on to outline what they actually did

[lucidrains]

does the following code look in line with what is described in the screenshot above?

[lucidrains]

@Sanqiang you mean "Iterative", not "Interactive"?

[Minami-su]

Your code is correct.

In fact, the author is suggesting that this self-reward method is similar to PCO, as described in this paper: https://arxiv.org/pdf/2312.16682.pdf.