[jenkins] ha-functional-testing-weekly All tests (build 310) completed with 136 failures

136/141 Stages failed: - ocpv centos10 zstream:no upstream:main tests:all: https://haci.fast.eng.rdu2.dc.redhat.com/job/global/job/ha-functional-testin... - pacemaker,resource,FilesystemGroup (nodes: 2) - pacemaker,usecase,ha-apache (nodes: 2) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 2) - pcs,cli,ClusterSetupParametrized (nodes: 3) - pcs,cli,CVE-2018-1079 (nodes: 3) - pcs,cli,NodeStandby (nodes: 3) - pcs,cli,Properties (nodes: 3) - pcs,cli,ResourceCreate (nodes: 3) - pcs,cli,rhbz1380372 (nodes: 3) - pcs,cli,Stonith (nodes: 3) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 3) - pacemaker,resource,FilesystemGroup (nodes: 3) - pacemaker,usecase,ha-apache (nodes: 3) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 3) - pcs,cli,CheckpointDiff (nodes: 3) - pcs,cli,ClusterConfigUpdate (nodes: 3) - pcs,cli,DefaultsSet (nodes: 3) - pcs,cli,RemoteGuestNodes (nodes: 3) - pcs,cli,remote_node,remote_node_invalid_cases (nodes: 3) - pcs,cli,rename,rename_cluster (nodes: 3) - pcs,cli,resource,description (nodes: 3) - pcs,cli,rhbz1328870 (nodes: 3) - pcs,cli,rhbz1362493 (nodes: 3) - pcs,cli,rhbz1419661 (nodes: 3) - pcs,cli,rhbz1443418 (nodes: 3) - pcs,cli,rhbz1502715 (nodes: 3) - pcs,cli,rhbz1568353 (nodes: 3) - pcs,cli,SetupRandomPorts (nodes: 3) - pcs,cli,Timeouts (nodes: 3) - pacemaker,recovery,RecoverySwitchFailure (nodes: 3) - pacemaker,recovery,RecoveryNodeNetworkFailure (nodes: 3) - pacemaker,recovery,RecoveryRandomMultiNode (nodes: 3) - pacemaker,recovery,RecoveryRandomMultiNodeNQ (nodes: 3) - pacemaker,recovery,RecoveryNodeNetworkFailure,sbd-only,network_disruptor (nodes: 3) - pacemaker,recovery,RecoveryNodeNetworkFailure,sbd-with-one-device,network_disruptor (nodes: 3) - pcs,cli,ResourceMove (nodes: 3) - setup (nodes: 4) - pacemaker,resource,IPv4,recovery (nodes: 4) - pacemaker,recovery,RecoveryActiveNode (nodes: 4) - pacemaker,recovery,RecoveryActiveNode,sbd-only,kill_reboot (nodes: 4) - pcs,cli,Auth (nodes: 4) - pcs,cli,ClusterCibConcurrentDiff (nodes: 4) - pcs,cli,ClusterCibPush (nodes: 4) - pcs,cli,ClusterSetupParametrized (nodes: 4) - pcs,cli,ClusterStartStop (nodes: 4) - pcs,cli,CVE-2018-1086 (nodes: 4) - pcs,cli,DaemonSanity (nodes: 4) - pcs,cli,export_config,alerts_export (nodes: 4) - pcs,cli,export_config,constraint_export (nodes: 4) - pcs,cli,export_config,nodes_attributes_export (nodes: 4) - pcs,cli,export_config,nodes_utilizations_export (nodes: 4) - pcs,cli,export_config,operation_defaults_export (nodes: 4) - pcs,cli,export_config,property_export (nodes: 4) - pcs,cli,export_config,resource_defaults_export (nodes: 4) - pcs,cli,export_config,resource_export (nodes: 4) - pcs,cli,export_config,stonith_export (nodes: 4) - pcs,cli,export_config,tags_export (nodes: 4) - pcs,cli,meta_validation,create_meta_primitive_resources (nodes: 4) - pcs,cli,meta_validation,resource_meta (nodes: 4) - pcs,cli,NodeAddRemove (nodes: 4) - pcs,cli,NodeMaintenance (nodes: 4) - pcs,cli,NodeStandby (nodes: 4) - pcs,cli,OperationDefaults (nodes: 4) - pcs,cli,pqc,pqc_certificates (nodes: 4) - pcs,cli,pqc,pqc_pcsd_server (nodes: 4) - pcs,cli,Properties (nodes: 4) - pcs,cli,ResourceCreate (nodes: 4) - pcs,cli,ResourceManageUnmanageMonitor (nodes: 4) - pcs,cli,ResourceSafeDisable (nodes: 4) - pcs,cli,rhbz1380372 (nodes: 4) - pcs,cli,rhbz1382004 (nodes: 4) - pcs,cli,rhbz1387106 (nodes: 4) - pcs,cli,rhbz1390609 (nodes: 4) - pcs,cli,rhbz1574898 (nodes: 4) - pcs,cli,status,status_query (nodes: 4) - pcs,cli,status,status_wait (nodes: 4) - pcs,cli,Stonith (nodes: 4) - pcs,cli,StonithLevel (nodes: 4) - pcs,cli,Tags (nodes: 4) - pacemaker,recovery,RecoveryRandomNode (nodes: 4) - pacemaker,recovery,RecoveryRestartPacemaker-AllNodes (nodes: 4) - pacemaker,recovery,RecoveryResourceFailure (nodes: 4) - pacemaker,resource,Apache,recovery (nodes: 4) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 4) - pacemaker,resource,Clone (nodes: 4) - pacemaker,resource,ConstraintsTest (nodes: 4) - pacemaker,resource,Group (nodes: 4) - pacemaker,resource,Filesystem (nodes: 4) - pacemaker,resource,FilesystemGroup (nodes: 4) - pacemaker,usecase,ha-apache (nodes: 4) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 4) - lvm,lvm_config_no-vdo,cluster-lvmlockd (nodes: 4) - pcs,cli,BackupRestore (nodes: 4) - pcs,cli,CheckpointDiff (nodes: 4) - pcs,cli,ClusterAuthkey (nodes: 4) - pcs,cli,ClusterConfigUpdate (nodes: 4) - pcs,cli,DefaultsSet (nodes: 4) - pcs,cli,meta_validation,defaults_set_meta (nodes: 4) - pcs,cli,meta_validation,remote_node_meta (nodes: 4) - pcs,cli,QuorumDevice (nodes: 4) - pcs,cli,RemoteGuestNodes (nodes: 4) - pcs,cli,remote_node,remote_node_add_remove (nodes: 4) - pcs,cli,remote_node,remote_node_invalid_cases (nodes: 4) - pcs,cli,remote_node,remote_node_valid_options (nodes: 4) - pcs,cli,rename,rename_cluster (nodes: 4) - pcs,cli,resource,description (nodes: 4) - pcs,cli,rhbz1328870 (nodes: 4) - pcs,cli,rhbz1362493 (nodes: 4) - pcs,cli,rhbz1419661 (nodes: 4) - pcs,cli,rhbz1443418 (nodes: 4) - pcs,cli,rhbz1502715 (nodes: 4) - pcs,cli,rhbz1568353 (nodes: 4) - pcs,cli,SetupRandomPorts (nodes: 4) - pcs,cli,Timeouts (nodes: 4) - pacemaker,recovery,RecoverySwitchFailure (nodes: 4) - pacemaker,recovery,RecoveryNodeNetworkFailure (nodes: 4) - pacemaker,recovery,RecoveryRandomMultiNode (nodes: 4) - pacemaker,recovery,RecoveryRandomMultiNodeNQ (nodes: 4) - pacemaker,recovery,RecoveryNodeNetworkFailure,sbd-only,network_disruptor (nodes: 4) - pacemaker,recovery,RecoveryNodeNetworkFailure,sbd-with-one-device,network_disruptor (nodes: 4) - pacemaker,usecase,apachewebfarm,haproxy (nodes: 4)

- ocpv centos10 zstream:no upstream:next-stable tests:all: https://haci.fast.eng.rdu2.dc.redhat.com/job/global/job/ha-functional-testin... - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 2) - pacemaker,resource,FilesystemGroup (nodes: 2) - pacemaker,usecase,ha-apache (nodes: 2) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 2) - pcs,cli,ClusterSetupParametrized (nodes: 3) - pcs,cli,CVE-2018-1079 (nodes: 3) - pcs,cli,NodeAddRemove (nodes: 3) - pcs,cli,NodeMaintenance (nodes: 3) - pcs,cli,ResourceCreate (nodes: 3) - pcs,cli,ResourceManageUnmanageMonitor (nodes: 3) - pcs,cli,rhbz1387106 (nodes: 3) - pcs,cli,rhbz1390609 (nodes: 3) - pcs,cli,Tags (nodes: 3) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 3) - pacemaker,resource,FilesystemGroup (nodes: 3) - pacemaker,usecase,ha-apache (nodes: 3) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 3) - pcs,cli,ClusterAuthkey (nodes: 3) - pcs,cli,ClusterConfigUpdate (nodes: 3) - pcs,cli,DefaultsSet (nodes: 3) - pcs,cli,QuorumDevice (nodes: 3) - pcs,cli,RemoteGuestNodes (nodes: 3) - pcs,cli,rhbz1328870 (nodes: 3) - pcs,cli,rhbz1443418 (nodes: 3) - pcs,cli,rhbz1502715 (nodes: 3) - pcs,cli,ClusterCibConcurrentDiff (nodes: 4) - pcs,cli,ClusterCibPush (nodes: 4) - pcs,cli,ClusterSetupParametrized (nodes: 4) - pcs,cli,ClusterStartStop (nodes: 4) - pcs,cli,CVE-2018-1079 (nodes: 4) - pcs,cli,CVE-2018-1086 (nodes: 4) - pcs,cli,NodeAddRemove (nodes: 4) - pcs,cli,OperationDefaults (nodes: 4) - pcs,cli,Properties (nodes: 4) - pcs,cli,ResourceCreate (nodes: 4) - pcs,cli,ResourceManageUnmanageMonitor (nodes: 4) - pcs,cli,ResourceSafeDisable (nodes: 4) - pcs,cli,rhbz1382004 (nodes: 4) - pcs,cli,rhbz1387106 (nodes: 4) - pcs,cli,rhbz1390609 (nodes: 4) - pcs,cli,Stonith (nodes: 4) - pcs,cli,StonithLevel (nodes: 4) - pcs,cli,Tags (nodes: 4) - pacemaker,recovery,RecoveryRandomNode (nodes: 4) - pacemaker,recovery,RecoveryRestartPacemaker-AllNodes (nodes: 4) - pacemaker,recovery,RecoveryResourceFailure (nodes: 4) - pacemaker,resource,Apache,recovery (nodes: 4) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 4) - pacemaker,resource,Clone (nodes: 4) - pacemaker,resource,ConstraintsTest (nodes: 4) - pacemaker,resource,Group (nodes: 4) - pacemaker,resource,Filesystem (nodes: 4) - pacemaker,resource,FilesystemGroup (nodes: 4) - pacemaker,usecase,ha-apache (nodes: 4) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 4) - lvm,lvm_config_no-vdo,cluster-lvmlockd (nodes: 4) - pcs,cli,BackupRestore (nodes: 4) - pcs,cli,ClusterAuthkey (nodes: 4) - pcs,cli,ClusterConfigUpdate (nodes: 4) - pcs,cli,DefaultsSet (nodes: 4) - pcs,cli,QuorumDevice (nodes: 4) - pcs,cli,RemoteGuestNodes (nodes: 4) - pcs,cli,rhbz1328870 (nodes: 4) - pcs,cli,rhbz1419661 (nodes: 4) - pcs,cli,rhbz1502715 (nodes: 4) - pcs,cli,SetupRandomPorts (nodes: 4) - pcs,cli,Timeouts (nodes: 4) - pacemaker,usecase,apachewebfarm,haproxy (nodes: 4)

- ocpv centos10 zstream:no upstream:none tests:all: https://haci.fast.eng.rdu2.dc.redhat.com/job/global/job/ha-functional-testin... - pcs,cli,ClusterStartStop (nodes: 2) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 2) - pacemaker,resource,FilesystemGroup (nodes: 2) - pacemaker,usecase,ha-apache (nodes: 2) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 2) - pcs,cli,ClusterSetupParametrized (nodes: 3) - pcs,cli,StonithLevel (nodes: 3) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 3) - pacemaker,resource,FilesystemGroup (nodes: 3) - pacemaker,usecase,ha-apache (nodes: 3) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 3) - pcs,cli,CheckpointDiff (nodes: 3) - pcs,cli,RemoteGuestNodes (nodes: 3) - pcs,cli,rhbz1328870 (nodes: 3) - pcs,cli,SetupRandomPorts (nodes: 3) - pcs,cli,Auth (nodes: 4) - pcs,cli,ClusterCibConcurrentDiff (nodes: 4) - pcs,cli,ClusterSetupParametrized (nodes: 4) - pcs,cli,ClusterStartStop (nodes: 4) - pcs,cli,CVE-2018-1086 (nodes: 4) - pcs,cli,NodeAddRemove (nodes: 4) - pcs,cli,NodeMaintenance (nodes: 4) - pcs,cli,NodeStandby (nodes: 4) - pcs,cli,Properties (nodes: 4) - pcs,cli,ResourceCreate (nodes: 4) - pcs,cli,ResourceSafeDisable (nodes: 4) - pcs,cli,rhbz1380372 (nodes: 4) - pcs,cli,rhbz1382004 (nodes: 4) - pcs,cli,rhbz1387106 (nodes: 4) - pcs,cli,rhbz1574898 (nodes: 4) - pcs,cli,status,status_query (nodes: 4) - pcs,cli,status,status_wait (nodes: 4) - pcs,cli,Stonith (nodes: 4) - pcs,cli,Tags (nodes: 4) - pacemaker,recovery,RecoveryRandomNode (nodes: 4) - pacemaker,recovery,RecoveryRestartPacemaker-AllNodes (nodes: 4) - pacemaker,recovery,RecoveryResourceFailure (nodes: 4) - pacemaker,resource,Apache,recovery (nodes: 4) - pacemaker,resource,Apache,variant:latency,latency:300ms (nodes: 4) - pacemaker,resource,Clone (nodes: 4) - pacemaker,resource,ConstraintsTest (nodes: 4) - pacemaker,resource,Group (nodes: 4) - pacemaker,resource,Filesystem (nodes: 4) - pacemaker,resource,FilesystemGroup (nodes: 4) - pacemaker,usecase,ha-apache (nodes: 4) - pacemaker,usecase,NFS_Server-AP,lvmlockd,exclusive (nodes: 4) - lvm,lvm_config_no-vdo,cluster-lvmlockd (nodes: 4) - pcs,cli,CheckpointDiff (nodes: 4) - pcs,cli,ClusterConfigUpdate (nodes: 4) - pcs,cli,DefaultsSet (nodes: 4) - pcs,cli,QuorumDevice (nodes: 4) - pcs,cli,RemoteGuestNodes (nodes: 4) - pcs,cli,rhbz1328870 (nodes: 4) - pcs,cli,rhbz1419661 (nodes: 4) - pcs,cli,rhbz1502715 (nodes: 4) - pcs,cli,rhbz1568353 (nodes: 4) - pacemaker,usecase,apachewebfarm,haproxy (nodes: 4)

Run reason: Started by upstream project "global/all-weekly" build number 53 Total runtime: 3 days 16 hr

Full log: https://ci.kronosnet.org/job/global/job/ha-functional-testing-weekly/310/con...