31/138 Stages failed: - aws rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7751/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7635/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7651/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7730/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7725/ - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7699/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- azure rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7634/ failed in deploy_cluster

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7650/ failed in deploy_cluster

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7722/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7681/ failed in create_cluster

- gcp rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7679/ failed in create_cluster

- gcp rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7675/ failed in create_cluster

- gcp rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7677/ failed in create_cluster

- gcp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7682/ failed in create_cluster

- gcp rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7680/ failed in create_cluster

- gcp rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7676/ failed in create_cluster

- gcp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7678/ failed in create_cluster

- ibmvpc rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7639/ - setup (nodes: 4)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7765/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt centos10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7762/ failed in deploy_cluster

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7753/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7755/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7711/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7712/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7769/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7760/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7761/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7723/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7724/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7741/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7743/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 590 Total runtime: 14 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/267/con...