Fix Segformer test for num_classes 1 with softmax#2401

sampathweb wants to merge 2 commits intokeras-team:masterkeras-team/keras-cv:masterfrom

sampathweb:fix-keras-cv-test-segformersampathweb/keras-cv:fix-keras-cv-test-segformerCopy head branch name to clipboard