[jenkins] ha-functional-testing-weekly (build 262) completed with 49 failures

49/138 Stages failed: - aliyun rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7160/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7166/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7127/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7093/ failed in deploy_cluster

- aliyun rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7132/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7182/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7164/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7186/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7176/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7165/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7069/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7143/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7138/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7098/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7111/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7144/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7104/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- azure rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7181/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7185/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7076/ failed in deploy_cluster

- azure rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7146/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7148/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7192/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7194/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7191/ - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7062/ - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7078/ - cleanup (nodes: 3)

- gcp rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7086/ - lvm,snapper,display_snap,nostack,singlenode (nodes: 1)

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7163/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7103/ - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7115/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7167/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7161/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- ibmvpc rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7065/ - setup (nodes: 4)

- ibmvpc rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7125/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7135/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7092/ - setup (nodes: 4)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7193/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7170/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7171/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7130/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7131/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7195/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7177/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7178/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7137/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7139/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7156/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7157/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 586 Total runtime: 14 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/262/con...