[jenkins] ha-functional-testing-weekly (build 264) completed with 49 failures

49/138 Stages failed: - aliyun rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7440/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7441/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7408/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aliyun rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7411/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7465/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7466/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7455/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7353/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7422/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7419/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7384/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7428/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- aws rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7421/ - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7382/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7388/ - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- azure rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7461/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7433/

- azure rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7464/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7352/ failed in deploy_cluster

- azure rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7424/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- azure rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7430/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7463/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7469/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7338/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7354/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 4)

- gcp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7346/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7362/ - pacemaker,resource,IPv4,recovery (nodes: 3)

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7426/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7438/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7410/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- ibmvpc rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7402/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7407/ - setup (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7470/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7446/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7448/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7405/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7406/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7471/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7452/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7453/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7415/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7417/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7335/

- osp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7351/

- osp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7343/ failed in create_cluster

- osp rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7359/ failed in create_cluster

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7416/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7409/ failed in create_cluster

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/7429/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 588 Total runtime: 15 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/264/con...