setNxtBlk Plugin (Debug Version) << NOTE >> o This version is an enhanced version of ../v0/. It was used to isolate a QM bug whose behavior is described below ("Bug"). o The main differences with the non-debug version: - addition of xdata[] for logging ring_out data - control message to get xdata[] to the user - the 'go' variable indicates when to record ring_out data and is also used to limit the amount of data collected (19 pkts). By default 'go' is 0. When the user enters the 'go' control message, 'go' is set to 1 which enables ring_out data logging to xdata[] and is incremented once for each ring_out recording. When 'go' reaches its limit (e.g., 20), recording stops. Typically, I toggled 'go' after the test sequence: ... send 3 ping pkts from n1p0 to n1p1 next= PLUGIN1 ... send 1 Mbps UDP traffic from n1p0 to n1p1 next= QM go # begin recording ... send 1 Mbps UDP traffic from n1p0 to n1p1 If you want to resume recording, you will need to toggle 'go' twice (once to zero the value of 'go' and turn off recording, and once to set the value of 'go' to 1 and enable recording). << Function >> o Counts and forwards pkts to the block specified by dlNextBlk where the user can specify the value of dlNextBlk through control messages. o The user actually specifies the next block value using the following string values: User Specifies dlNextBlk Actual Value (external) (internal) ---------------------------------------------------- PLUGIN0 PACKET_IN_RING_0 6 PLUGIN1 PACKET_IN_RING_1 7 PLUGIN2 PACKET_IN_RING_2 8 PLUGIN3 PACKET_IN_RING_3 9 PLUGIN4 PACKET_IN_RING_4 10 MUX MUX 1 QM QM 0 DROP DROP 7 ; i.e., Freelist Manager << Control Messages >> Type Command Proto Semantics Example set next= set dlNextBlock IN: "next= PLUGIN2" OUT: "PLUGIN2" get =vers display version number IN: "=vers" OUT: "2.0" =npkts display npkts IN: "=npkts" OUT: "37124" =next display value of dlNextBlock IN: "=next" (external string form) OUT: "QM" =raw display value of dlNextBlock IN: "=raw" (internal value) OUT: "QM" =xdata get next 2 values of xdata[] IN: "=xdata" misc reset reset counters for all queues IN: "reset" OUT: "OK" debug toggle debug_on (init 0) IN: "debug" OUT: debug_on go toggle ring_out recording IN: "go" << Plugin Counters >> o Counter 0: number of pkts received by plugin o Counter 4: error count << Tested Behavior >> o Configuration - 1 NPR, hosts n1p0 and n1p1, setNxtBlk plugins on MEs 0-3, mycount plugin on ME 4 (see setNxtBlk.exp) - Plugin was made with 5 packet handling threads - Monitor plugin counter 0 (pkt counters) on all 5 MEs o Preliminaries - Send 'd' msg to mycount to turn off debugging o Traffic for test 1 - 3 ping pkts from n1p0 to n1p1 (see ~kenw/bin/setNB script) o Traffic for test 2 - 1 Mbps UDP traffic from n1p0 to n1p1 (modify ~kenw/bin/setNB script) produces about 87 pkts n1p0> ping -c 3 n1p1 # send 3 ping pkts ... see ME 0 counter incrementing ... Send 'next= 4' msg to ME 0 n1p0> ping -c 3 n1p1 # send 3 ping pkts ... see ME 0 and ME 4 counter incrementing ... Send 'next= 1' msg to ME 0 Send 'next= 2' msg to ME 1 Send 'next= 3' msg to ME 2 Send 'next= 4' msg to ME 3 n1p0> ping -c 3 n1p1 # send 3 ping pkts ... see MEs 0-4 counters incrementing ... o Test 2 (chain setNxtBlk plugins together: setNxtBlk ME k sends to ME k+1) - Repeat for UDP traffic n1p1> iperf -s -u -w 4m # server w/ 4 MB rcv buffer n1p0> iperf -c n1p1 -b Xm # send at X Mbps for 10 sec - Drop results (setNxtBlk+drops.exp) + 'netstat -s | grep -i err' to find errors detected by endhost + Includes monitoring of drop counters (registers 24-28, 30-31, 33-34; i.e., HEC, iplen, hlen, vers, plc-plugin, descr, QM, rx, tx) Sending #Drops/Pkts Sent Note Rate (Mbps) ------------------------------------------------------------- 100 0/85471 (4 times) no non-zero drop counters & no endhost errors 200 0/172414 (4 times) ditto 400 0/344828 (4 times) ditto 800 0/714286 (4 times) ditto 1000 56/813915 (1 time) 14 TX drops, 0 endhost errors 56/813917 (1 time) 14 TX drops, 0 endhost errors 56/813914 (1 time) 14 TX drops, 0 endhost errors 1200 + Here is another test when I used 'iperf -s -u -w 1m' (1 MB rcv buffer instead of 4 MB ... there were no endhost errors reported but there seemed to be a few drops that couldn't be accounted for) Sending #Drops/Pkts Sent Note Rate (Mbps) ------------------------------------------------------------- 10 0/ repeated many times 100 1/85471 (1 time) no non-zero drop counters 0/85471 (4 times) & no endhost errors 200 1/172414 (1 time) ditto 400 1/344828 (8 times) ditto 0/344828 (4 times) ditto 600 1/526316 (2 times) ditto 0/526316 (2 times) ditto 800 3/714286 (1 time) ditto 1/714286 (2 times) ditto 1000 56/813916 (1 time) 14 TX drops, 0 endhost errors 55/813932 (1 time) 13 TX drops, 0 endhost errors 58/813915 (1 time) 14 TX drops, 0 endhost errors 1100 184910/999985 (1 time) 16 TX drops, 0 endhost errors port 1.1 rate = 1200 Mbps << Bug >> o The following sequence sometimes produces incorrect behavior for ME 0: ME 0 'next= QM ' send 3 ping pkts // OK ME 0 'next= PLUGIN1' send 3 ping pkts // OK ME 0 'next= QM ' send 3 ping pkts // pkts don't get out of TX ME 0 'next= PLUGIN1' send 3 ping pkts // OK again ME 0 'next= DROP' send 3 ping pkts // OK ME 0 'next= PLUGIN1' send 3 ping pkts // OK again Seems to be ok sometimes if just ping traffic, but problem always shows up with 1 Mbps UDP traffic This behavior is not serious because we normally want to change the forwarding to some block other than QM. We rarely want to change it back to QM. Notes: 1) helper_check_meta() can be activated by turning on debugging but it never finds a bad field value when pkts don't get out of TX. I'm not sure what happened to the pkts. 2) Pkts to anything other than QM always make it to their destination. The same incorrect behavior occurs on all of the other 'setNxtBlk' instances which are loaded into MEs 0-4. o Right now, helper_set_meta_default() doesn't set the rightmost 3 bits of the uc_mc_bits field; i.e., it is 0 when the meta-pkt arrives at the next plugin. None of the plugin framework code uses it ... so, is it really a bug?