[jenkins] ha-functional-testing-weekly (build 272) completed with 44 failures

44/138 Stages failed: - aliyun rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8428/ failed in deploy_cluster

- aliyun rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8420/ failed in deploy_cluster

- aliyun rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8416/ failed in deploy_cluster

- aliyun rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8349/ failed in deploy_cluster

- aliyun rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8409/ failed in deploy_cluster

- aliyun rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8377/ failed in deploy_cluster

- aws rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8372/ failed in Final delete_cluster

- aws rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8370/ failed in Final delete_cluster

- aws rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8368/ failed in Final delete_cluster

- aws rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8373/ failed in Final delete_cluster

- aws rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8371/ failed in Final delete_cluster

- aws rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8369/ failed in Final delete_cluster

- aws rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8331/ failed in Final delete_cluster

- aws rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8347/ failed in Final delete_cluster

- aws rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8339/ failed in Final delete_cluster

- aws rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8355/ failed in Final delete_cluster

- aws rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8366/ failed in Final delete_cluster

- aws rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8364/ failed in Final delete_cluster

- aws rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8360/ failed in Final delete_cluster

- aws rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8362/ failed in Final delete_cluster

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8367/ failed in Final delete_cluster

- aws rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8365/ failed in Final delete_cluster

- aws rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8361/ failed in Final delete_cluster

- aws rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8363/ failed in Final delete_cluster

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8346/ failed in deploy_cluster

- gcp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8348/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8460/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt centos10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8457/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8447/ failed in deploy_cluster

- libvirt rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8438/ failed in deploy_cluster

- libvirt rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8431/ failed in deploy_cluster

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8448/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8432/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8412/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8413/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8463/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8453/ failed in deploy_cluster

- ocpv rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8445/ failed in deploy_cluster

- ocpv rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8439/ failed in deploy_cluster

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8454/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8423/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8424/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8442/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8443/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 594 Total runtime: 12 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/272/con...