Add the pretraining data creation tool #1310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

lshpku wants to merge 1 commit into PaddlePaddle:develop from lshpku:pretrain-data

+509 −0

lshpku commented Oct 23, 2025 •

edited

Loading

迁移PaddleNLP的预训练数据转换工具

原脚本：PaddleNLP/llm/tools/preprocess/create_pretraining_data.py

原使用文档：https://paddlenlp.readthedocs.io/en/latest/llm/dataset.html

其实只改了from ernie.tokenizer import Ernie4_5_Tokenizer这一行，如果不是因为PaddleNLP不再支持EB模型，直接用PaddleNLP里面的脚本也可以

paddle-bot bot commented Oct 23, 2025

Thanks for your contribution!


          Add the pretraining data creation tool

235494f

lshpku force-pushed the pretrain-data branch from 3c6c009 to 235494f Compare

October 23, 2025 09:26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet