Commits

commits@lists.kronosnet.org

1 participants
19593 discussions

[kronosnet/kronosnet] 0f67ee: [links] stabilize latency calculation when nodes a...

by Fabio M. Di Nitto

Branch: refs/heads/master Home: https://github.com/kronosnet/kronosnet Commit: 0f67ee86745d52d68f376c92e96e1dd6661e9f5d https://github.com/kronosnet/kronosnet/commit/0f67ee86745d52d68f376c92e96e1… Author: Fabio M. Di Nitto <fdinitto(a)redhat.com> Date: 2019-09-06 (Fri, 06 Sep 2019) Changed paths: M libknet/threads_heartbeat.c M libknet/threads_rx.c Log Message: ----------- [links] stabilize latency calculation when nodes are not responsive The following scenario is more of a corner case than normal, but this change allows to better deal with this situation: 1) 2 nodes cluster (corosync) (node A and node B) 2) kill -stop $(pidof corosync) on node A 3) node B will continue to send ping packets to node A 4) node A is accumulating those ping packets in the kernel network socket 5) wait some seconds and unpause node A 6) node A will start processing the ping packets in the queue and send pong replies to node B 7) node B will see an extreme increase of latency due those "obsoleted" ping/pong packets 8) node B, as latency increases, will take longer and longer to notice that node A is down due to the pong_timeout adjustment for latency (required for initial cluster spike). the solution: 1) Use average latency to calculate pong_timeout_adj vs latency_max. Averate latency will go down again in time, while latency_max is never reset. 2) RX thread will filter out all pong packets that have higher latency than currently configure pong_timeout. This barrier should have been in place even before. this solution reduces the latency spike on node B to a perfectly reasonable level and it will all eventually stabilize over time as latency samples increase and latency will reduce. Please be aware that using a pong_timeout smaller than latency will simply mark the link down now. Signed-off-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> Commit: 28e2b563e8acb0ac0eeb7a4c39efc6e7bf54ec53 https://github.com/kronosnet/kronosnet/commit/28e2b563e8acb0ac0eeb7a4c39efc… Author: Fabio M. Di Nitto <fdinitto(a)redhat.com> Date: 2019-09-09 (Mon, 09 Sep 2019) Changed paths: M libknet/threads_heartbeat.c M libknet/threads_rx.c Log Message: ----------- Merge pull request #251 from kronosnet/latency-fixes [links] stabilize latency calculation when nodes are not responsive Compare: https://github.com/kronosnet/kronosnet/compare/512e433b0b3d...28e2b563e8ac

5 years

knet-build-all-nonvoting - Build # 1230 - Failure!

by jenkins＠kronosnet.org

knet-build-all-nonvoting - Build # 1230 - Failure: Check console output at https://ci.kronosnet.org/job/knet-build-all-nonvoting/1230/ to view the results.

5 years

[kronosnet/kronosnet]

by kronosnet CI bot

Branch: refs/heads/coverity_scan Home: https://github.com/kronosnet/kronosnet

5 years

[kronosnet/kronosnet] 0f67ee: [links] stabilize latency calculation when nodes a...

by Fabio M. Di Nitto

Branch: refs/heads/latency-fixes Home: https://github.com/kronosnet/kronosnet Commit: 0f67ee86745d52d68f376c92e96e1dd6661e9f5d https://github.com/kronosnet/kronosnet/commit/0f67ee86745d52d68f376c92e96e1… Author: Fabio M. Di Nitto <fdinitto(a)redhat.com> Date: 2019-09-06 (Fri, 06 Sep 2019) Changed paths: M libknet/threads_heartbeat.c M libknet/threads_rx.c Log Message: ----------- [links] stabilize latency calculation when nodes are not responsive The following scenario is more of a corner case than normal, but this change allows to better deal with this situation: 1) 2 nodes cluster (corosync) (node A and node B) 2) kill -stop $(pidof corosync) on node A 3) node B will continue to send ping packets to node A 4) node A is accumulating those ping packets in the kernel network socket 5) wait some seconds and unpause node A 6) node A will start processing the ping packets in the queue and send pong replies to node B 7) node B will see an extreme increase of latency due those "obsoleted" ping/pong packets 8) node B, as latency increases, will take longer and longer to notice that node A is down due to the pong_timeout adjustment for latency (required for initial cluster spike). the solution: 1) Use average latency to calculate pong_timeout_adj vs latency_max. Averate latency will go down again in time, while latency_max is never reset. 2) RX thread will filter out all pong packets that have higher latency than currently configure pong_timeout. This barrier should have been in place even before. this solution reduces the latency spike on node B to a perfectly reasonable level and it will all eventually stabilize over time as latency samples increase and latency will reduce. Please be aware that using a pong_timeout smaller than latency will simply mark the link down now. Signed-off-by: Fabio M. Di Nitto <fdinitto(a)redhat.com>

5 years

sbd-build-all-nonvoting-clang - Build # 9 - Failure!

by jenkins＠kronosnet.org

sbd-build-all-nonvoting-clang - Build # 9 - Failure: Check console output at https://ci.kronosnet.org/job/sbd-build-all-nonvoting-clang/9/ to view the results.

5 years

sbd-build-all-nonvoting-clang - Build # 8 - Failure!

by jenkins＠kronosnet.org

sbd-build-all-nonvoting-clang - Build # 8 - Failure: Check console output at https://ci.kronosnet.org/job/sbd-build-all-nonvoting-clang/8/ to view the results.

5 years

sbd-build-all-nonvoting - Build # 20 - Failure!

by jenkins＠kronosnet.org

sbd-build-all-nonvoting - Build # 20 - Failure: Check console output at https://ci.kronosnet.org/job/sbd-build-all-nonvoting/20/ to view the results.

5 years

[kronosnet/kronosnet]

by kronosnet CI bot

Branch: refs/heads/coverity_scan Home: https://github.com/kronosnet/kronosnet

5 years

knet-build-all-nonvoting - Build # 1227 - Failure!

by jenkins＠kronosnet.org

knet-build-all-nonvoting - Build # 1227 - Failure: Check console output at https://ci.kronosnet.org/job/knet-build-all-nonvoting/1227/ to view the results.

5 years

resource-agents-build-all-voting - Build # 585 - Failure!

by jenkins＠kronosnet.org

resource-agents-build-all-voting - Build # 585 - Failure: Check console output at https://ci.kronosnet.org/job/resource-agents-build-all-voting/585/ to view the results.

5 years

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Commits