Integrating OmDet Turbo in Transformers 🤗

[yonigozlan]

Hi Om people!

I am an MLE at Hugging Face, and given the popularity and performance of your model, we wanted to see if you would be interested in working with us to integrate OmDet Turbo into the Transformers 🤗 library. Looking forward to hearing back from you!

Best,
Yoni

[hx621]

@yonigozlan
Hello Yoni! Thank you for reaching out and for your interest in integrating OmDet Turbo into the Transformers 🤗 library. We're excited about the possibility of collaborating with Hugging Face on this project. Let's discuss the details and next steps to make this happen.

[yonigozlan]

Hi @hx621, great to hear back from you!
The integration of OmDetTurbo is a current work in progress over here 🤗.
Please tell us if you spot anything wrongly implemented or any subtleties we have missed!
Thank you.
Yoni

[yonigozlan]

Hi again @hx621, @P3ngLiu. Does this model support batch inference with different labels and prompt for different images in the same batch?
I'm trying to add support for it in the Transformers implementation, however it seems like I'm getting inconsistent results when using labels list of different length in the same batch. Is this a known behavior of the model? It could also be a problem with the Transformers implementation.

[hx621]

Hi again @hx621, @P3ngLiu. Does this model support batch inference with different labels and prompt for different images in the same batch? I'm trying to add support for it in the Transformers implementation, however it seems like I'm getting inconsistent results when using labels list of different length in the same batch. Is this a known behavior of the model? It could also be a problem with the Transformers implementation.

@yonigozlan During training, each image within the same batch has different labels and prompts. Therefore, inference should also support different images with different labels, and I have already verified this. You can simply modify our demo to implement this feature。
a）provide different labels for each input image

OmDet/omdet/inference/det_engine.py

Line 72 in 542ce97

img['label_set'] = labels

b) Change here to labels unique to each image

OmDet/omdet/inference/det_engine.py

Line 85 in 542ce97

cls = flat_labels[int(cls)]

[yonigozlan]

Hi Om team! @hx621 @P3ngLiu
OmDet-Turbo is almost ready to be released on Transformers 🤗!
As a reminder, here is the PR.
Last step before releasing is to upload the Transformers version of the weights on the hub under your organization name.
What would be your preferred name? We were thinking of omlab/omdet-turbo-swin-tiny-hf. Indeed, since there is already a checkpoint uploaded for OmdetTurbo swin tiny under your organization name (omlab/OmDet-Turbo_tiny_SWIN_T), and if you would like to keep this one, it would be nice to have a "...-hf" suffix to the Transformers version of the weights.
Looking forward to your answer!

[P3ngLiu]

@yonigozlan
Thank you very much for integrating our model! We can't wait to see Omdet-Turbo on Transformers. The name omlab/omdet-turbo-swin-tiny-hf sounds good to us as well. Will you be uploading the model, or do we need to upload it ourselves?

[yonigozlan]

@P3ngLiu It might be simpler if we do it on our side since we'll have the weights ready to go if that's ok with you

[P3ngLiu]

@yonigozlan okok, you can do it.

[yonigozlan]

Hi again!
OmDet-Turbo should be released on Transformers tomorrow!
The Transformers weights are already available here.
Feel free to add or modify anything in the current model card!

[P3ngLiu]

@yonigozlan That's so great! Thank you again for your contribution in integrating OmDet-Turbo into Transformers. We are really happy to hear this news.

[yonigozlan]

It's live now here!

[P3ngLiu]

@yonigozlan
Hi Yoni！
Thank you once again for helping us integrate OmDet-Turbo into Transformers, which has significantly increased the attention our model has received. We plan to open-source a Base version of the model soon. The pre-training is expected to be completed within a week. Unlike the Tiny version, the Base version utilizes ConvNext Base as the vision backbone. We will first attempt to convert the weights to the Transformers version ourselves. If any code modifications are needed, could you provide assistance?

[yonigozlan]

Hi @P3ngLiu, very nice to hear that! And of course for the base model, don't hesitate to ping me if you have any questions or if there is a problem with the current implementation, I'll be happy to help.
Can't wait to see the performance of the base model!

[dnth]

Will there be any updates on the base model release? @P3ngLiu