Support SDAR

[grimoire]

Since we are refactoring chat template, we will use template of internlm2 for now.

from lmdeploy import pipeline, PytorchEngineConfig, GenerationConfig, ChatTemplateConfig


from lmdeploy.pytorch.tools.utils import Timer, visualize_pipe_out


if __name__ == '__main__':
    model_path = 'JetLM/SDAR-1.7B-Chat'
    chat_template_config = ChatTemplateConfig('internlm2')

    log_level = 'WARNING'

    dllm_unmasking_strategy='low_confidence_dynamic'
    # dllm_unmasking_strategy='sequential'

    prompts = [
        'hakuna matata!',
        'The quick brown fox jumps over the lazy dog.'
        ]

    backend_config = PytorchEngineConfig(
        tp=1,
        dllm_block_length=4,
        dllm_unmasking_strategy=dllm_unmasking_strategy,
    )

    gen_config = GenerationConfig(
        max_new_tokens=512,
    )


    with pipeline(model_path, backend_config=backend_config,
                  chat_template_config=chat_template_config, log_level=log_level) as pipe:
        outputs = pipe(prompts, gen_config=gen_config)
        print(outputs)

[lvhan028]

What's the purpose of prealloc_size? I see it's always passed as 0 from the schedule method. Is this intended for future use or should it be removed?

[grimoire]

Over allocate kv cache might be useful for future models.

[lvhan028]

Does SDAR conflict with --quant-policy fp8? I tried it, it repeated meaningless words.
FYI, --quant-policy for this model is not required in this PR

[lvhan028]

@zhulinJulia24 may put JetLM/SDAR-8B-Chat, JetLM/SDAR-30B-A3B-Chat into CI

[RunningLeon]

seq_length should be all max_q_seqlen, not all 1