Skip to content

[PERFSCALE-3943] LACP bonding for perf/scale lab (2) #706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

[PERFSCALE-3943] LACP bonding for perf/scale lab (2) #706

wants to merge 11 commits into from

Conversation

josecastillolema
Copy link
Member

No description provided.

@openshift-ci openshift-ci bot requested review from akrzos and jtaleric September 27, 2025 10:48
@josecastillolema
Use ocpinventory.json instead of foreman info
791715e 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Set bastion_bond0_interfaces in create-inventory
5af712e 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Fixed nmcli bond setup
168e431 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Added bastion bond interface configuration to both inventory templates
322b078 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
wait-hosts-discovered-set the hostname bonded scenario
b8a6406 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Add temporary debug task
1455831 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Fix bond port numbering
9f21222 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Temporary debug task
d13e9d4 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Fixed inventory generation
906ec48 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Copy link
Member Author

Ongoing testing in: openshift/release#69765
SNO deployments look fine ✔️

  • Bastion host
# nmcli con
NAME                 UUID                                  TYPE      DEVICE
Wired connection 1   d223b670-71e3-33b5-9532-c31f77fa811d  ethernet  eno1np0
eno2np1              2dfdcbe6-030d-4061-ac08-92979d58b9e7  ethernet  eno2np1
bond0                44949383-52ad-4e5f-be91-0d75e883cbe4  bond      bond0
bond0-slave-ens1f0   1d6eb689-7536-40d1-a13c-a07f18335a72  ethernet  ens1f0
bond0-slave-ens1f1   393074e4-ed46-4707-b500-477b2a925c19  ethernet  ens1f1
lo                   f3932fd2-b907-40ab-8855-af73dfee5efa  loopback  lo
bond0-slave-eno1np0  db04bb01-eca6-48c5-a5a4-37adb2fa0782  ethernet  --
Wired connection 2   4e92f0b5-fed6-3fd1-b8a0-0e22075fbc3f  ethernet  --
Wired connection 3   ee2fb421-83bd-3b2f-8450-245250de4f9a  ethernet  --
Wired connection 4   f5429206-1761-3586-a64b-577b9898c596  ethernet  --
  • SNO host
sh-5.1# ovs-vsctl show
b6e31319-37dc-4594-8b44-df31e8576371
    Bridge br-ex
        Port patch-br-ex_f19-h02-000-r640-to-br-int
            Interface patch-br-ex_f19-h02-000-r640-to-br-int
                type: patch
                options: {peer=patch-br-int-to-br-ex_f19-h02-000-r640}
        Port br-ex
            Interface br-ex
                type: internal
        Port bond0
            Interface bond0
                type: system

sh-5.1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.14.0-427.92.1.el9_4.x86_64

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 40:a6:b7:2a:74:40
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 21
        Partner Key: 12
        Partner Mac Address: 84:c1:c1:47:04:fc

Slave Interface: ens1f0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2a:74:40
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 40:a6:b7:2a:74:40
    port key: 21
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: 84:c1:c1:47:04:fc
    oper key: 12
    port priority: 127
    port number: 12
    port state: 63

Slave Interface: ens1f1
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2a:74:41
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 40:a6:b7:2a:74:40
    port key: 21
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: 84:c1:c1:47:04:fc
    oper key: 12
    port priority: 127
    port number: 6
    port state: 63

@josecastillolema
Fix MNO bonding mac assignment logic
09503de 
Signed-off-by: Jose Castillo Lema <[email protected]>
@josecastillolema
Copy link
Member Author

And bonding working with MNO as well:

# oc get node
NAME               STATUS   ROLES                  AGE   VERSION
f19-h0x-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h0y-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h0z-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h1x-000-r640   Ready    worker                 22h   v1.31.13
f19-h1y-000-r640   Ready    worker                 22h   v1.31.13

# oc debug node/f19-h17-000-r640
Starting pod/f19-h17-000-r640-debug-ngkgf ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.216.8
If you don't see a command prompt, try pressing enter.
sh-5.1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.14.0-427.92.1.el9_4.x86_64

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: xx
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 21
        Partner Key: 15
        Partner Mac Address: xx

Slave Interface: ens1f0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2b:a2:90
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: xx
    port key: 21
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: xx
    oper key: 15
    port priority: 127
    port number: 3
    port state: 63

Slave Interface: ens1f1
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: xx
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: xx
    port key: 21
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: xx
    oper key: 15
    port priority: 127
    port number: 9
    port state: 63

sh-5.1# chroot /host
sh-5.1# ovs-vsctl show
...
    Bridge br-ex
        Port patch-br-ex_f19-h17-000-r640-to-br-int
            Interface patch-br-ex_f19-h17-000-r640-to-br-int
                type: patch
                options: {peer=patch-br-int-to-br-ex_f19-h17-000-r640}
        Port br-ex
            Interface br-ex
                type: internal
        Port bond0
            Interface bond0
                type: system
    ovs_version: "3.5.2-33.el9fdp"

@josecastillolema
Copy link
Member Author

/assign @akrzos

@josecastillolema
Copy link
Member Author

@akrzos this is ready for review! Thanks

@josecastillolema josecastillolema added the enhancement New feature or request label Oct 2, 2025
akrzos
akrzos requested changes Oct 3, 2025
View reviewed changes
Copy link
Member

@akrzos akrzos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is a slight mis-understanding to the indexes of mac addresses provided by the scale lab and the nic names saved in hw_nic_name that we should address.

ansible/roles/bastion-network/tasks/main.yml
loop:
- "{{ bastion_bond0_interface1 }}"
- "{{ bastion_bond0_interface2 }}"
ignore_errors: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anytime we put an ignore_errors on a task, we should try to include a comment as to why it exists. I realize I had missed this the first time. Could you add a short one line comment similar to something like this. Ideally we don't end up ignoring an error that we want to know about, but I do understand in some situations ignore_errors becomes the best option.

ansible/roles/create-inventory/defaults/main/networks.yml
bond0_port1: 1
bond0_port2: 2
# Default behavior uses interfaces at indices 0 and 1 from hw_nic_name mapping
bond0_port1: 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused at choosing "index" 0 since that would refer to the public nic in hw_nic_name Is that intentional?

ansible/roles/create-inventory/tasks/main.yml
- name: MNO - Set controlplane0 bond mac addresses
set_fact:
controlplane0_bond0_macs: "{{ (controlplane0_foreman_data.json.interfaces | selectattr('name', 'eq', bond0_interface1) | first).mac }},{{ (controlplane0_foreman_data.json.interfaces | selectattr('name', 'eq', bond0_interface2) | first).mac }}"
controlplane0_bond0_macs: "{{ ocpinventory.json.nodes[1].mac[bond0_port1|int] }},{{ ocpinventory.json.nodes[1].mac[bond0_port2|int] }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For mac addresses, the scale lab does not include the public nic in the list of mac addresses so this is likely the reason you may have adjusted bond0_port1 and bond0_port2 back one index. The issue here is the scale lab index and hw_nic_name do not match for mac and nic names. This was intentional at the time hw_nic_name because it was solely for managing what nic name would be choosen for hypervisors.

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Oct 3, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from akrzos. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akrzos akrzos akrzos requested changes

@jtaleric jtaleric Awaiting requested review from jtaleric

Assignees

@akrzos akrzos

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@josecastillolema @akrzos