[jenkins] ha-functional-testing-weekly (build 274) completed with 34 failures

34/138 Stages failed: - aliyun rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8638/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8727/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8607/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8652/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8656/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8622/ failed in deploy_cluster

- gcp rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8739/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8737/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8741/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8738/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8731/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8708/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8679/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8653/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8712/ failed in pacemaker,resource,IPv4,recovery

- gcp rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8682/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8654/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8611/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8736/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8722/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8697/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8723/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8705/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- libvirt rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8699/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8676/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8677/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8740/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8724/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8725/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8684/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8686/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8701/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

- osp rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8666/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3)

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8721/ - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 597 Total runtime: 18 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/274/con...