Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions api/orchestration/v1alpha1/groupversion_info.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
const (
StormServiceKind = "StormService"
RoleSetKind = "RoleSet"
PodSetKind = "PodSet"
)

var (
Expand Down
3 changes: 3 additions & 0 deletions api/orchestration/v1alpha1/podset_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ type PodSetSpec struct {
// Stateful indicates whether pods should have stable network identities
// +optional
Stateful bool `json:"stateful,omitempty"`

// +optional
SchedulingStrategy *SchedulingStrategy `json:"schedulingStrategy,omitempty"`
}

// PodSetStatus defines the observed state of PodSet
Expand Down
93 changes: 90 additions & 3 deletions api/orchestration/v1alpha1/roleset_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,96 @@ type RoleSetSpec struct {
UpdateStrategy RoleSetUpdateStrategyType `json:"updateStrategy,omitempty"`

// +optional
SchedulingStrategy SchedulingStrategy `json:"schedulingStrategy,omitempty"`
SchedulingStrategy *SchedulingStrategy `json:"schedulingStrategy,omitempty"`
}

// +enum
// +kubebuilder:validation:MaxProperties=1
type SchedulingStrategy struct {
PodGroup *schedv1alpha1.PodGroupSpec `json:"podGroup,omitempty"`
GodelSchedulingStrategy *GodelSchedulingStrategySpec `json:"godelSchedulingStrategy,omitempty"`

CoschedulingSchedulingStrategy *CoschedulingSchedulingStrategySpec `json:"coschedulingSchedulingStrategy,omitempty"`

VolcanoSchedulingStrategy *VolcanoSchedulingStrategySpec `json:"volcanoSchedulingStrategy,omitempty"`
Comment on lines +43 to +47
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding these dependencies, do the strategies require these components or CRDs to run? For example, does the VolcanoSchedulingStrategy require the Volcano component to be installed to take effect? ​​Also, do we need to integrate with Aibrix (is that a large dependency)? Or how can we check if the CRD exists?

Copy link
Collaborator

@googs1025 googs1025 Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine that if a user sets VolcanoSchedulingStrategy but doesn't have Volcano installed. What will the final be? Is there a fallback mechanism?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we check if prd exist before each podgroup create/delete (with dynamic client), and if crd do not exist we just stop there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative is to add a feature gate that requires user to choose the scheduler implementation if they want to enable podgroup, therefore we can check prd exist at start time and just crash and complain if it does not (if crd is determined at start time we can add it to informer which is a plus).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like a feature gate approach is a good option. Regarding the CRD check, I understand we can this check crd when starting controller. If fg=true, users should install the corresponding component(like volcano...). We could implement this similarly.
FYI: https://github.com/Jeffwan/aibrix/blob/1008ec3260f0a85f573c45d0a6487080f10758b1/pkg/controller/controller.go#L62-L73

Also, if pg creation fails, we could throw an appropriate log or event to notify the user.

Copy link
Contributor Author

@Epsilon314 Epsilon314 Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tradeoff is

  1. add a feature-gate
    pro: since crd is known at start time, we can add it to schema therefore watching and modification will be easier;
    cons: since there is no obvious default value users have to config it explicitly, thus is not enabled out-of-box
  2. detect dynamically
    pro: config free, take effect immediately after scheduler is installed
    cons: hard to watch podgroup change; more apicall
    after rethink i feel like adding a feature gate would be a better choice , and we can simplify the added configuration complexity in release part for example some helm tricks

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2. after rethink i feel like adding a feature gate would be a better choice , and we can simplify the added configuration complexity in release part for example some helm tricks

FYI:
InftyAI/llmaz#316 (comment)

}

// GodelSchedulingStrategySpec uses godel scheduler's podgroup definition
type GodelSchedulingStrategySpec struct {
// MinMember defines the minimal number of members/tasks to run the pod group;
// if there's not enough resources to start all tasks, the scheduler
// will not start anyone.
MinMember int32 `json:"minMember,omitempty"`

// If specified, indicates the PodGroup's priority. "system-node-critical" and
// "system-cluster-critical" are two special reserved keywords which indicate the highest priorities.
// Any other name must be defined by creating a PriorityClass object with that name.
// If not specified, the PodGroup priority will be default.
// If default priority class doesn't exist, the PodGroup priority will be zero.
// +optional
PriorityClassName string `json:"priorityClassName,omitempty"`

// ScheduleTimeoutSeconds defines the maximal time of tasks to wait before run the pod group;
// +optional
ScheduleTimeoutSeconds *int32 `json:"scheduleTimeoutSeconds,omitempty"`

// Application indicates the podGroup belongs to a logical Application
// This will be used for coordinate with features like drf and faire share.
// +optional
Application string `json:"application,omitempty"`

// Affinity shows the affinity/anti-affinity rules that scheduler needs to follow
// when scheduling instances of this pod group.
// +optional
Affinity *schedv1alpha1.Affinity `json:"affinity,omitempty"`
}

// CoschedulingSchedulingStrategySpec uses coscheduling scheduler-plugin's podgroup definition
type CoschedulingSchedulingStrategySpec struct {
// MinMember defines the minimal number of members/tasks to run the pod group;
// if there's not enough resources to start all tasks, the scheduler
// will not start any.
// The minimum is 1
// +kubebuilder:validation:Minimum=1
MinMember int32 `json:"minMember,omitempty"`

// MinResources defines the minimal resource of members/tasks to run the pod group;
// if there's not enough resources to start all tasks, the scheduler
// will not start any.
MinResources v1.ResourceList `json:"minResources,omitempty"`

// ScheduleTimeoutSeconds defines the maximal time of members/tasks to wait before run the pod group;
ScheduleTimeoutSeconds *int32 `json:"scheduleTimeoutSeconds,omitempty"`
}

// VolcanoSchedulingStrategySpec uses volcano's podgroup definition
type VolcanoSchedulingStrategySpec struct {
// MinMember defines the minimal number of members/tasks to run the pod group;
// if there's not enough resources to start all tasks, the scheduler
// will not start anyone.
MinMember int32 `json:"minMember,omitempty" protobuf:"bytes,1,opt,name=minMember"`

// MinTaskMember defines the minimal number of pods to run each task in the pod group;
// if there's not enough resources to start each task, the scheduler
// will not start anyone.
MinTaskMember map[string]int32 `json:"minTaskMember,omitempty" protobuf:"bytes,1,opt,name=minTaskMember"`

// Queue defines the queue to allocate resource for PodGroup; if queue does not exist,
// the PodGroup will not be scheduled. Defaults to `default` Queue with the lowest weight.
// +optional
Queue string `json:"queue,omitempty" protobuf:"bytes,2,opt,name=queue"`

// If specified, indicates the PodGroup's priority. "system-node-critical" and
// "system-cluster-critical" are two special keywords which indicate the
// highest priorities with the former being the highest priority. Any other
// name must be defined by creating a PriorityClass object with that name.
// If not specified, the PodGroup priority will be default or zero if there is no
// default.
// +optional
PriorityClassName string `json:"priorityClassName,omitempty" protobuf:"bytes,3,opt,name=priorityClassName"`

// MinResources defines the minimal resource of members/tasks to run the pod group;
// if there's not enough resources to start all tasks, the scheduler
// will not start anyone.
MinResources *v1.ResourceList `json:"minResources,omitempty" protobuf:"bytes,4,opt,name=minResources"`
}

// +enum
Expand Down Expand Up @@ -94,6 +178,9 @@ type RoleSpec struct {
// DisruptionTolerance indicates how many pods can be unavailable during the preemption/eviction.
// +optional
DisruptionTolerance DisruptionTolerance `json:"disruptionTolerance,omitempty"`

// +optional
SchedulingStrategy *SchedulingStrategy `json:"schedulingStrategy,omitempty"`
}

// +enum
Expand Down
120 changes: 115 additions & 5 deletions api/orchestration/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading