Hey Chrissie,
i found the bugs in totemknet that made knet so slow and now I am
getting up to 50% more cpgbench performance (19MB/sec on 64 bytes pckts)
vs udpu (13MB/sec).
First bug is the dst_host_filter. this is not your fault, your thinking
was sound but it didn't do the right thing. We need to fix the knet API
doc and possibly I can also fix the code to be smarter.
The issue is that for RX packet, you make an exception where if we are
sending pckts to self, you return that the pckt is unicast. This is
confusing the packet deduplicator because it will try to match mcast
pckts against unicast seq num.
The dst_host_filter shouldn't change unicast/mcast, but return data from
the pckt. In theory this can be simplified because those information are
already inside the onwire packet.
I'll review that soon enough because it's confusing and incorrect.
Anyway to unblock testing:
+#if 0
if (tx_rx == KNET_NOTIFY_RX) {
dst_host_ids[0] = this_host_id;
*dst_host_ids_entries = 1;
res = 0; /* already home */
}
else {
+#endif
and remove the stray } ;)
Second is the MTU calculation that's generating unnecessary fragmented
pckts.
extern void totemknet_net_mtu_adjust (void *knet_context, struct
totem_config *totem_config)
{
- fprintf(stderr, "MTU = %d\n", totem_config->net_mtu);
- totem_config->net_mtu = 1444;
+ // fabbione: need to export libknet header size from libknet somehow
+ totem_config->net_mtu -= totemip_udpip_header_size(AF_INET) + 23;
}
This calculation is correct for AF_INET interfaces (we need to fix it
for IPv6 and such but it's a start.
This also unveiled a less important bug in knet that doesn't export all
MTU information to applications, but it doesn't affect corosync directly
(that hard coded 23 should be an API call).
Using the combination of those 2 gives me a 14 to 15 MB/sec on 64 bytes
packets in cpgbench using default netmtu of 1500.
The last boost is netmtu changes. Bumping netmtu in corosync.conf to
4096 I get to 19MB/sec.
Now there is a bug in corosync netmtu somewhere. I didn't bother to
investigate because it's sunday,
I tried 8192 and corosync started to misbehave, as if it's not sending
pckts till buffer is full (or that's the feeling I got).
Bumping netmtu to 64000 makes corosync hang.
So there is definitely some work to be done to fix it and most likely
get better performances out of the system.
Cheers
Fabio
