[jenkins] ha-functional-testing-weekly (build 259) completed with 59 failures

59/138 Stages failed: - aliyun rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6741/ failed in deploy_cluster

- aliyun rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6736/ failed in deploy_cluster

- aliyun rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6734/ failed in deploy_cluster

- aliyun rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6746/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aliyun rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6739/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aliyun rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6735/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6771/ failed in deploy_cluster

- aws rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6767/ failed in deploy_cluster

- aws rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6762/ failed in deploy_cluster

- aws rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6772/ failed in deploy_cluster

- aws rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6765/ failed in deploy_cluster

- aws rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6647/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6730/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6726/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6699/ - setup (nodes: 4)

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6733/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6727/ - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6704/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6662/ failed in deploy_cluster

- gcp rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6780/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6778/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6770/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6781/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6779/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6773/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6656/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2)

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6743/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1) - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6723/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6687/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1)

- gcp rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6696/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6756/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6729/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6707/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1) - lvm,raid_sanity,display_raid,raid4,singlenode (nodes: 1) - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6651/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6667/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6659/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6675/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6757/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6750/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6708/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6710/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6759/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6751/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6709/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6712/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6744/ failed in deploy_cluster

- libvirt rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6740/ failed in deploy_cluster

- libvirt rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6737/ failed in deploy_cluster

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6755/ failed in deploy_cluster

- ocpv rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6753/ failed in deploy_cluster

- ocpv rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6748/ failed in deploy_cluster

- osp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6645/ failed in create_cluster

- osp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6661/ failed in create_cluster

- osp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6653/ failed in create_cluster

- osp rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6669/ failed in create_cluster

- osp rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6680/ failed in create_cluster

- osp rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6676/ failed in create_cluster

- osp rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6677/ failed in create_cluster

- osp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/6679/ failed in create_cluster

Run reason: Started by upstream project "global/all-weekly/default" build number 583 Total runtime: 13 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/259/con...