[Dynamic batch] Investigate refactoring opportunities for batch management in Plugin and Compiler

[DariaMityagina]

Closed as a duplicate for - #32350

Details:

• The task is to explore the possibility of moving the reshaping process to the plugin. This would allow the compiler to receive either a network with batch size 1 if reshaping is successful, or a network with a non-1 batch size if reshaping fails. This approach aims to simplify the batch handling process and reduce dependencies on the compiler.

The concept in this PR approach:

1. Verify if the model is compatible with batching on the plugin side.
2. Attempt to reshape the model using set_batch(1).
3. If this is successful, we continue with the PLUGIN batch. The reshaped model is sent to the compiler, and afterward, we adjust the metadata post-compilation to retain the original batch value.

Need to modify the logic to be aligned with:

1. Verify if the model is compatible with batching on the plugin side.
2. Attempt to reshape the model using set_batch(1).
3. If this is successful, we continue with the PLUGIN batch.
4. Write/Read batch size into NPU plugin metadata (write when compiling a model, read when importing a model).
5. The reshaped model is sent to the compiler. BATCH_MODE must be set to COMPILER.
6. Adjust the metadata post-compilation to retain the original batch value - To keep backward/forward compatibility, create/update blob metadata with what you have in the plugin metadata. Create it in case of CiD, update it in case of MLIR.

Tickets:

• E-176749

[DariaMityagina]

Maybe keep originalShapesMap to avoid using [0].

[DariaMityagina]

This section is necessary to preserve the original batch information. After reshaping the model in lines 660-676 and compiling it in line 736, the metadata will reflect shapeFromIRModel as the reshaped version, rather than the original.

[DariaMityagina]

Points to consider: Is it possible to avoid altering the metadata? Can we eliminate dependence on it when dealing with dynamic batch scenarios?

[DariaMityagina]

TODO: check static batching as well.

[sivanov-work]

set_batch() is naive and cannot proceed sometimes especially where batch dimetion is not specified in layout information. I'd recommend to add an attempt to do debatchDynamicModel in case of exception

[sivanov-work]

What if we set not originalBatch but an entire originalShape? to do not speculate with an actual BATCH_AXIS position, which may not be represented by 0 index

[DariaMityagina]

Yes, there was such an idea: #31691 (comment)
Thanks!

[sivanov-work]

I'd propose to extend NetworkMetadata class by aggregating additional layout information or better off introducing a new class alike PluginNetworkMetadata which will hold NetworkMetadata and layouts as well.

The purpose of adding this layout is to let user specify it so that we will stick to it instead of speculate with BATCH_AXIS position which is not equal to 0 in the generic case as we had ensured in previous PRs

[DariaMityagina]

Suggested change

	for (auto& in : networkMeta.inputs) {
	if (in.shapeFromIRModel.has_value() && batchSize.has_value()) {
	in.shapeFromIRModel.value()[intel_npu::utils::BATCH_AXIS] = ov::Dimension(1, batchSize.value());
	}
	}
	for (auto& out : networkMeta.outputs) {
	if (out.shapeFromIRModel.has_value() && batchSize.has_value()) {
	out.shapeFromIRModel.value()[intel_npu::utils::BATCH_AXIS] = ov::Dimension(1, batchSize.value());
	}
	}
	if (batchSize.has_value()) {
	for (auto& in : networkMeta.inputs) {
	if (in.shapeFromIRModel.has_value()) {
	in.shapeFromIRModel.value()[intel_npu::utils::BATCH_AXIS] = ov::Dimension(1, batchSize.value());
	}
	}
	for (auto& out : networkMeta.outputs) {
	if (out.shapeFromIRModel.has_value()) {
	out.shapeFromIRModel.value()[intel_npu::utils::BATCH_AXIS] = ov::Dimension(1, batchSize.value());
	}
	}
	}