[DLStreamer] gvagenai: enable memory mapper, add support for prompt-path #938
Conversation
yunowo
requested review from
tbujewsk,
OskarFiedot,
marcin-wadolkowski,
nszczygl9,
dmichalo,
msmiatac and
a team
as code owners
yunowo
changed the title
| auto gst_frame = std::make_shared<dlstreamer::GSTFrame>(buffer, info); | ||
| auto mapped_frame = mapper->map(gst_frame, dlstreamer::AccessMode::Read); | ||
|
|
||
| // Convert to Mat, code from gvawatermark |
There was a problem hiding this comment.
Conversion of VAMemory / DMABuf to OpenCV Mat will involve memory copy.
Please add support for remote tensors instead.
There was a problem hiding this comment.
RemoteTensor is not supported. See CVS-160602
There was a problem hiding this comment.
Is GPU memory not supported at all, or the limitation is lack of ov::preprocessor in genai front-end?
We should be able to generate RGB or BGR images in VAMemory, and create tensor wrappers around.
There was a problem hiding this comment.
Both. VAAPI surface doesn't support 24 bits RGB or BGR, only supports 32 bits RGBA or BGRA.
For GenAI, the image encoding (preprocessing) handles the data on CPU. See classes.cpp llava_image_embed_make_with_bytes_slice and resample functions.
Also the VLMPipeline class doesn't expose RemoteContext, also RemoteContext cannot be passed into VLMPipeline, so no way to create a RemoteTensor.
Description
Enabled memory mapper for gvagenai so it supports VAMemory, VASurface, DMABuf frames now.
Add support for prompt-path.
Any Newly Introduced Dependencies
None
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist: