[tests] clean the pipeline level group offloading tests#12330

sayakpaul wants to merge 4 commits intomainhuggingface/diffusers:mainfrom

sanitize-pipe-go-testshuggingface/diffusers:sanitize-pipe-go-testsCopy head branch name to clipboard