62/138 Stages failed: - aliyun rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8254/ failed in create_cluster

- aliyun rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8252/ failed in create_cluster

- aliyun rhel10 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8250/ failed in create_cluster

- aliyun rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8255/ failed in create_cluster

- aliyun rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8253/ failed in create_cluster

- aliyun rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8251/ failed in create_cluster

- aliyun rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8189/

- aliyun rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8205/

- aliyun rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8197/

- aliyun rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8213/ aborted in Final delete_cluster aborted in Final delete_cluster aborted in Final delete_cluster aborted in Final delete_cluster

- aliyun rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8248/ failed in create_cluster

- aliyun rhel9 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8246/ failed in create_cluster

- aliyun rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8249/ failed in create_cluster

- aliyun rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8247/ failed in create_cluster

- aliyun rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8242/ failed in create_cluster

- aws rhel10 zstream:no upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8294/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8315/ - pacemaker,resource,IPv4,recovery (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 4)

- aws rhel10 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8293/ - pacemaker,resource,IPv4,recovery (nodes: 2)

- aws rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8187/

- aws rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8203/

- aws rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8195/ failed in pacemaker,recovery,RecoveryActiveNode failed in cleanup

- aws rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8211/

- aws rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8279/ - setup (nodes: 2) - pacemaker,resource,IPv4,recovery (nodes: 3)

- aws rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8245/ - pacemaker,resource,IPv4,recovery (nodes: 4)

- azure rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8186/

- azure rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8202/ failed in deploy_cluster

- azure rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8194/

- azure rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8210/ failed in pacemaker,resource,IPv4,recovery failed in setup failed in cleanup failed in cleanup failed in pacemaker,resource,IPv4,recovery

- gcp rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8311/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8316/ - pacemaker,recovery,RecoveryActiveNode (nodes: 2) - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel10 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8306/ - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- gcp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8188/

- gcp rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8204/

- gcp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8196/

- gcp rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8212/ failed in cleanup failed in cleanup failed in pacemaker,resource,IPv4,recovery failed in setup failed in setup failed in pacemaker,recovery,RecoveryActiveNode

- gcp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8281/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel9 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8234/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel9 zstream:yes upstream:next-stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8272/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3)

- gcp rhel9 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8244/ - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- ibmvpc rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8207/

- ibmvpc rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8199/

- ibmvpc rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8215/ failed in cleanup

- ibmvpc rhel9 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8232/ - pacemaker,recovery,RecoveryActiveNode (nodes: 4)

- libvirt centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8320/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8303/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8304/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8192/ failed in pacemaker,resource,IPv4,recovery

- libvirt rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8265/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- libvirt rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8266/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv centos10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8321/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8309/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel10 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8310/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8190/

- ocpv rhel8 zstream:no upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8206/ aborted in Final delete_cluster

- ocpv rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8198/ failed in cleanup

- ocpv rhel8 zstream:yes upstream:stable tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8214/ aborted in Final delete_cluster

- ocpv rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8273/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- ocpv rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8276/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel8 zstream:no upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8185/ failed in lvm,snapper,display_snap,nostack,singlenode

- osp rhel8 zstream:yes upstream:none tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8193/ aborted in Final delete_cluster

- osp rhel9 zstream:no upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8287/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

- osp rhel9 zstream:yes upstream:main tests:smoke: https://ci.kronosnet.org/job/global/job/ha-functional-testing/8288/ - setup,setup_sbd,sbd-only (nodes: 2) - setup (nodes: 3) - pacemaker,resource,IPv4,recovery (nodes: 3) - pacemaker,recovery,RecoveryActiveNode (nodes: 3) - setup,setup_sbd,sbd-only (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - setup,setup_sbd,sbd-only (nodes: 4)

Run reason: Started by upstream project "global/all-weekly/default" build number 593 Total runtime: 14 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/271/con...