Fix: vNPU HAMi mode may not work with multi-type NPUs cluster

[kiritoxkiriko]

What type of PR is this?

BugFix

What this PR does / why we need it:

vNPU HAMi mode may not work with multi-type NPUs cluster

Which issue(s) this PR fixes:

Fixes #4778

Special notes for your reviewer:

According to the problem described in this issue #4778
I address this bug should related with IgnoredDevicesList, since the ${card_type}-memory resource should be ignored in scheduling.

After checking out the codes, I found out the old IgnoredDevicesList.Set method will refresh all of device in the list, which will be used for setting node resource

// pkg/scheduler/api/node_info.go:345
// setNodeOthersResource initialize sharable devices
func (ni *NodeInfo) setNodeOthersResource(node *v1.Node) {
	if node == nil {
		klog.Warningf("received argument of nil node, no need to set other resources for %s", ni.Name)
		return
	}

	ni.Others[gpushare.DeviceName] = gpushare.NewGPUDevices(ni.Name, node)
	ni.Others[vgpu.DeviceName] = vgpu.NewGPUDevices(ni.Name, node)
	ni.Others[vnpu.DeviceName] = vnpu.NewNPUDevices(ni.Name, node)
	ascend_ignored_list := []string{}
	for device_name, devices := range hami.NewAscendDevices(ni.Name, node) {
		ni.Others[device_name] = devices
		// only Devices for this node will add to ignore list
		ascend_ignored_list = append(ascend_ignored_list, devices.GetIgnoredDevices()...)
	}
	klog.V(5).Infof("ascend_ignored_list is %v", ascend_ignored_list)
	// IgnoredDevices will be set here,
	IgnoredDevicesList.Set(
		ni.Others[gpushare.DeviceName].(Devices).GetIgnoredDevices(),
		ni.Others[vgpu.DeviceName].(Devices).GetIgnoredDevices(),
		ni.Others[vnpu.DeviceName].(Devices).GetIgnoredDevices(),
		ascend_ignored_list,
	)
	klog.V(5).Infof("ignoredDevicesList is %v", IgnoredDevicesList.List())
}

// pkg/scheduler/api/shared_device_pool.go:99

var IgnoredDevicesList = ignoredDevicesList{}

type ignoredDevicesList struct {
	sync.RWMutex
	ignoredDevices []string
}

// this method will set and refresh the list
func (l *ignoredDevicesList) Set(deviceLists ...[]string) {
	l.Lock()
	defer l.Unlock()
	l.ignoredDevices = l.ignoredDevices[:0]
	for _, devices := range deviceLists {
		l.ignoredDevices = append(l.ignoredDevices, devices...)
	}
}

Only one node's device will be ignore here, and we may have multiple nodes, but the Set method will refresh the list very time it calls, so the ignore list will only keep last evaluated node's device in the end. That's why only one kind of NPU mem will be ignore in this case.

I tried to keep the orignial IgnoredDevicesList functions, and add new methods in this commit, and also use map to keep the list items unique since we don't want add duplicated resource in this list

Please take a look @archlitchi

Does this PR introduce a user-facing change?

NONE

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign shinytang6 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[volcano-sh-bot]

Welcome @kiritoxkiriko! It looks like this is your first PR to volcano-sh/volcano 🎉

[gemini-code-assist]

Summary of Changes

Hello @kiritoxkiriko, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug impacting the vNPU HAMi mode's functionality within clusters containing multiple NPU types. The core problem stemmed from the IgnoredDevicesList mechanism, which was designed to manage devices that should be excluded from scheduling. Previously, this list was being reset for each node processed, leading to an incomplete and incorrect set of ignored devices. The changes introduce a robust solution by refactoring the IgnoredDevicesList to properly aggregate and maintain a unique collection of ignored devices across all nodes, thereby ensuring correct resource scheduling in diverse NPU environments.

Highlights

• Bug Fix for Multi-Type NPU Clusters: Resolved an issue where vNPU HAMi mode failed in multi-type NPU clusters due to incorrect handling of ignored devices.
• Improved IgnoredDevicesList Management: The IgnoredDevicesList now correctly accumulates ignored devices from all nodes using new Append and AppendList methods, replacing the problematic Set method that would overwrite the list.
• Ensured Device Uniqueness: A new ignoredIndex map has been introduced within ignoredDevicesList to guarantee that each ignored device is stored only once, preventing duplicates.
• Updated Node Resource Setting: The setNodeOthersResource function in node_info.go has been updated to utilize the new AppendList method for adding ignored devices, ensuring proper aggregation.
• Enhanced Test Coverage: Existing unit tests for ignoredDevicesList have been updated to reflect the new initialization pattern using newIgnoredDevicesList().

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request fixes a bug where IgnoredDevicesList was being overwritten for each node in a multi-type NPU cluster, causing only the last node's devices to be considered. The fix correctly changes the logic to append devices from all nodes.

My review focuses on the correctness and maintainability of the new implementation. I've identified a potential issue where the global IgnoredDevicesList can accumulate stale entries, and I've suggested a strategy to address this. Additionally, I've provided a recommendation to improve the efficiency of the new AppendList method and a suggestion to add unit tests for the new functionality to ensure its robustness. Overall, the change is in the right direction to fix the reported bug.

[Copilot]

Pull request overview

This PR fixes a critical bug in vNPU HAMi mode where the scheduler failed to properly ignore device-specific resources in multi-type NPU clusters. The issue occurred because IgnoredDevicesList.Set() was overwriting the list on each node evaluation instead of accumulating device types across all nodes. The fix introduces Append() and AppendList() methods that accumulate unique device types using a map-based deduplication strategy.

Key changes:

• Added ignoredIndex map to ignoredDevicesList for efficient deduplication
• Modified Set() to deduplicate devices and filter empty strings
• Added Append(), AppendList(), and List() methods for accumulation-based operations
• Changed node_info.go to use AppendList() instead of Set() when registering ignored devices

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
pkg/scheduler/api/shared_device_pool.go	Core fix: added deduplication logic and new append methods for accumulating ignored device types across nodes
pkg/scheduler/api/node_info.go	Changed from Set() to AppendList() to accumulate devices from all nodes; added debug logging
pkg/scheduler/api/shared_device_pool_test.go	Updated tests to use new newIgnoredDevicesList() constructor

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[archlitchi]

please sign-off your commit to pass the DCO check

[DSFans2014]

ignored_list includes all devices, not just ascend.