REQUEST: Single GPU inference code for audio_driven model

[lovebeatz]

Could you release/guide on writing single GPU (high/low VRAM) inference code for audio-driven model? I assume loading the audio model using sample_gpu_poor.py with torch.load(strict="false") won't make sense at all. I tried converting the code from sample_batch.py for the audio-driven model while talking sample_gpu_poor.py as reference, while being assisted by Sonnet 4. The code exectuted correctly, the generation took the expected time, but the output had nothing convincing.

[zhouzhengguang]

Thanks for your interest. I think run_single_audio is what you need?

[lovebeatz]

#!/bin/bash
JOBS_DIR=$(dirname $(dirname "$0"))
export PYTHONPATH=${JOBS_DIR}:$PYTHONPATH
export MODEL_BASE=${JOBS_DIR}"/models"
export NCCL_DEBUG=OFF
checkpoint_path=${MODEL_BASE}"/hunyuancustom_audio_720P/mp_rank_00_model_states_fp8.pt"
current_time=$(date "+%Y.%m.%d-%H.%M.%S")
modelname='Tencent_HunyuanCustom_Audio_720P'
OUTPUT_BASEPATH=./results/${modelname}/${current_time}

export CPU_OFFLOAD=1
CUDA_VISIBLE_DEVICES=0 python3 hymm_sp/sample_gpu_poor.py
--audio-condition
--ref-image './assets/images/Kendrick.png'
--input-audio './assets/audios/kendrick2.wav'
--audio-strength 0.8
--pos-prompt "High-contrast, gritty realism. Shot from a low angle, a rapper delivers intense bars, leaning forward on a cracked leather couch in a stark, minimalist basement. A single bare bulb overhead creates dramatic chiaroscuro lighting, casting deep shadows. His face is etched with raw emotion and determination. Film grain, monochrome tones, dynamic energy, hyper-detailed textures."
--neg-prompt "Clean, bright, cheerful, colorful, studio lighting, relaxed, smiling, poor composition, awkward hand gestures, generic background, two persons, female, overexposed, distortion, blurring, subtitles, watermark, perfect teeth, plastic skin."
--ckpt ${checkpoint_path}
--video-size 720 1280
--sample-n-frames 129
--cfg-scale 7.5
--seed 1024
--infer-steps 36
--use-deepcache 1
--flow-shift-eval-video 13.0
--save-path ${OUTPUT_BASEPATH}
--use-fp8
--cpu-offload

this code

with this image as reference

gave this as result

Kendrick_audio2.mp4

[lovebeatz]

Thanks for your interest. I think run_single_audio is what you need?

would you like to shed some light please?