Unable to Profile DMATask on NPU40XX

[ColorsWind]

I am currently investigating the time breakdown of tasks within a single model. However, I've encountered an issue where I cannot profile the DMATask. I've spent a week trying to identify any missing information but have yet to resolve the problem. Below are the minimal steps to reproduce the issue. The platform is an Intel ultra258v laptop with openvino-nightly and numpy installed.

Step 1: Create the Sigmoid Model

import numpy as np
import openvino as ov
from openvino.runtime import op, opset1

n_len = 128
ov_type = ov.Type.f16

A = op.Parameter(ov_type, ov.Shape([n_len]))
C = opset1.sigmoid(A)
model = ov.Model(C, [A])
ov.save_model(model, "sigmoid.xml")

Step 2: Compile the Sigmoid Model with DMA Profiling Flags

vpux-translate --vpu-arch=NPU40XX \
    --vpux-profiling \
    --mlir-print-debuginfo \
    --import-IE sigmoid.xml -o sigmoid.mlir
    
vpux-opt --vpu-arch=NPU40XX \
    --default-hw-mode="profiling=true dma-profiling=true" \
    --lower-VPUIP-to-ELF sigmoid.mlir \
    -o sigmoid_out.mlir
vpux-translate --vpu-arch=NPU40XX --export-ELF sigmoid_out.mlir -o sigmoid.blob

Step 3: Run the Model

import os
os.environ['ZE_INTEL_NPU_LOGLEVEL'] = 'ERROR'

import openvino as ov
import numpy as np

core = ov.Core()
core.set_property('NPU', {
    'PERF_COUNT': True,
})
with open('sigmoid.blob', 'rb') as f:
    blob = f.read()
model = core.import_model(blob, device_name='NPU')
req = model.create_infer_request()
req.infer(np.random.random(128).astype(np.float16))
prof_info = req.profiling_info[0]
print('status', prof_info.status)
print('real_time', prof_info.real_time)
print('cpu_time', prof_info.cpu_time)
print('node_name', prof_info.node_name)
print('exec_type', prof_info.exec_type)
print('node_type', prof_info.node_type)

You will encounter the following error:

NPU_LOG: ERROR [compiler.cpp:289] Failed to get decoded profiling data in compiler

In fact, when I try using the Level Zero API to submit tasks directly and then use the Level Zero NPU extension API to get profiling results, I find that the DMA profiling output data is zero, while the DPU and SW-Kernel profiling outputs are valid. I am wondering how I can obtain the DMA task profiling results.

[ColorsWind]

I sent this issue to the OpenVINO repository.
openvinotoolkit/openvino#28285

[Kepontry]

Hi, @ColorsWind. How do you get the vpux-translate. I tried to set the ov_option(ENABLE_DEVELOPER_BUILD "Enable developer build with extra validation/logging functionality" ON), but the vpux-translate is not found in the results. I built this repo by using cmake -B build -S . -DENABLE_NPU_COMPILER_BUILD=ON, following the instructions in https://github.com/intel/linux-npu-driver

[ColorsWind]

Hi @Kepontry. I can share my build script with you:

export OPENVINO_HOME=/home/user/openvino
export NPU_PLUGIN_HOME=/home/user/npu_compiler
cd $OPENVINO_HOME
mkdir build
cd build
cmake \
    -D CMAKE_VERBOSE_MAKEFILE=OFF \
    -D ENABLE_DEVELOPER_BUILD=ON \
    -D CMAKE_BUILD_TYPE=Release \
    -D ENABLE_MLIR_COMPILER=ON \
    -D BUILD_SHARED_LIBS=ON \
    -D OPENVINO_EXTRA_MODULES=$NPU_PLUGIN_HOME \
    -D ENABLE_LTO=ON \
    -D ENABLE_FASTER_BUILD=ON \
    -D ENABLE_CPPLINT=OFF \
    -D ENABLE_TESTS=OFF \
    -D ENABLE_FUNCTIONAL_TESTS=OFF \
    -D ENABLE_SAMPLES=OFF \
    -D ENABLE_JS=OFF \
    -D ENABLE_PYTHON=OFF \
    -D ENABLE_PYTHON_PACKAGING=OFF \
    -D ENABLE_WHEEL=OFF \
    -D ENABLE_OV_ONNX_FRONTEND=ON \
    -D ENABLE_OV_PYTORCH_FRONTEND=ON \
    -D ENABLE_OV_PADDLE_FRONTEND=OFF \
    -D ENABLE_OV_TF_FRONTEND=OFF \
    -D ENABLE_OV_TF_LITE_FRONTEND=OFF \
    -D ENABLE_OV_JAX_FRONTEND=OFF \
    -D ENABLE_OV_IR_FRONTEND=ON \
    -D THREADING=TBB \
    -D ENABLE_TBBBIND_2_5=OFF \
    -D ENABLE_SYSTEM_TBB=OFF \
    -D ENABLE_TBB_RELEASE_ONLY=OFF \
    -D ENABLE_HETERO=OFF \
    -D ENABLE_MULTI=OFF \
    -D ENABLE_AUTO=OFF \
    -D ENABLE_AUTO_BATCH=OFF \
    -D ENABLE_TEMPLATE=OFF \
    -D ENABLE_PROXY=OFF \
    -D ENABLE_INTEL_CPU=OFF \
    -D ENABLE_INTEL_GPU=OFF \
    -D ENABLE_NPU_PLUGIN_ENGINE=ON \
    -D ENABLE_ZEROAPI_BACKEND=OFF \
    -D ENABLE_DRIVER_COMPILER_ADAPTER=OFF \
    -D ENABLE_INTEL_NPU_INTERNAL=OFF \
    -D ENABLE_INTEL_NPU_PROTOPIPE=OFF \
    -D BUILD_COMPILER_FOR_DRIVER=ON \
    -D ENABLE_PRIVATE_TESTS=OFF \
    -D ENABLE_NPU_LSP_SERVER=ON \
    ..
cmake --build . --config Release --target vpux-opt vpux-translate -j16

You will find vpux-translate in $OPENVINO_HOME\bin.

[Kepontry]

Hi, @ColorsWind . Thanks for your assistance, I found that the target vpux-translate depends on npu_translate_utils_static. However, the BUILD_SHARED_LIBS and ENABLE_MLIR_COMPILER flags should both be set to enable the target.

if(BUILD_SHARED_LIBS AND ENABLE_MLIR_COMPILER)
    add_subdirectory(vpux_translate_utils)
endif()

After manually setting BUILD_SHARED_LIBS=ON, I was able to build the vpux-translate. I also noticed that in the UD2025.12 version, the dependency is removed.

By the way, I also reproduced the issue on an Intel Ultra 258v system using OpenVINO built from the December 4th version.

I also opened an issue in the driver repo about the metrics used for profiling. It seems that only the first metric in the NOC metric groups works and can be used for DMA bandwidth monitoring. Btw, could you share what the DPU-related metrics mentioned earlier look like and how to collect them? I couldn't find any relevant documentation.

[ColorsWind]

@Kepontry

The hardware outputs a buffer as profiling output, with some bits related to the execution times of DPU, DMA, and SHAVE tasks. By parsing these bits, you can determine the start and end times of each task. The issue I discovered is that all bits related to DMA are zero, which seems to be a bug in either the hardware or firmware.

To collect this information, you can directly use the higher-level interfaces of OpenVINO. If you want to obtain more low-level information, you can use the Level Zero interface. However, I found that in the underlying buffer, all bits related to DMA are zero, meaning it is fundamentally impossible to obtain the execution times of DMA tasks. I spent a month on this issue and it seems unsolvable. Finally, I switched to the Ultra100 series hardware. It seems that this issue does not exist on some older platforms.

But if you are only concerned with hardware bandwidth, I recommend using some indirect methods. For example, you can construct a bandwidth-bound kernel (such as vector add) and record its execution time. Another method is to use VTune, which can collect some bandwidth-related information. (By the way, you need to use VTune on Windows; when I tested it on Linux two months ago, it did not support the functionality I mentioned earlier.)

[Kepontry]

That helps a lot, thanks! @ColorsWind

[Maxim-Doronin]

Hello @Kepontry @ColorsWind! We’ll do our best to bring back vpux-translate in the upcoming UD18 release, along with the relevant tests and documentation.
@DariaMityagina could you please take a look at the issue reported in the sub?