The ONL Tutorial

Tutorial >> Examples

TOC

Writing A Plugin

Applications

• 10 Mbps link emulator
• With high-low priority packet scheduler

Learn

• Structure of a simple plugin
Developing a plugin
• Strategy
• Gotchas
• Debug messages
• Outputing debug messages
• Compiling for debugging
• Hints

Preparation (REWRITE ===============================)

• 1 NSP with 2 extra hosts and loop between ports 6 and 7
• Traffic sources
  • n1p1a: ping to n1p3
  • n1p1b: 20 Mbps UDP to n1p4
  • n1p2: TCP to n1p5
• Traffic sinks at n1p4 (UDP) and n1p5 (TCP)
• All pkts go out port 7 and into port 6 where they get routed. Return traffic takes reverse path.
• Run central control script from onlusr

The table below shows the shell scripts used to start traffic senders and receivers in this example. Before beginning this example, you should copy the files to the directory on the ONL user host where you store your executables and shell scripts (e.g., ~/bin/). You can do that by either selecting the each link below to get the files and then copying them to the appropriate place. Alternatively, you can copy them directly from the directory /users/onl/export/Examples/Writing_A_Plugin/ after SSHing to ONL user host onl.arl.wustl.edu.

This example has four parts. We begin with a basic experiment involving two UDP flows passing through a bottleneck link at port 7 and then show three variations involving packet scheduling, TCP traffic, and packet delay emulation. It assumes that you have already created the SSH tunnel required by the RLI and started the RLI on your client host.

1. Two UDP flows sharing a single queue at port 7

   Packets from UDP flows contend for the bottleneck output link in a FIFO manner.

2. Two UDP flows using separate queues at port 7

   Packets from two UDP flows are placed in separate queues where each flow gets their share of the output link capacity. The user can specify the shares through the Queue Table at port 7. Separating the flows also smooths out the traffic bandwidth obtained by each UDP flow.

3. Two TCP flows using separate queues at port 7

   We switch to the TCP traffic scripts and show how TCP shows bandwidth behavior that is characteristic of TCP's congestion avoidance algorithm.

4. Delay TCP ACK packets at port 6

   We delay the TCP ACK packets by installing a delay plugin and an egress GM filter at port 6.

Steps (In Brief)

Here are the major steps in this example:

1. Commit a 1-NSP dumbbell configuration
2. Setup the route tables at ports 2-5
3. Install an egress GM filter at port 7 to direct all packets to a single queue
4. Set the link bandwidth at port 7
5. Configure egress queue 300 at port 7
6. Create the bandwidth chart
7. Create the queue length chart
8. Create the packet drop chart
9. Start the two UDP receivers
10. Start the two UDP senders
11. Stop the UDP senders

Steps (In Detail)

1. Commit a 1-NSP dumbbell configuration

   Create the 1-NSP dumbbell topology shown on the The_Remote_Laboratory_Interface page.



• Select Topology => Add Cluster.
• Select Topology => Add Link.
• Drag the cursor with the left mouse button from port 7 to port 6.
• Select File => Commit.

  When done, the links will become solid and the host icons will turn to a dark blue.



2. Setup the route tables at ports 2-7

Ports	Next Hop
2, 3	7
4, 5	6
6, 7	Default Forwarding

Traffic from the sources at ports 2 and 3 should go to port 7 where it loops back into port 6. From port 6, traffic goes to the appropriate destination (port 4 or 5). Return TCP traffic (ACK packets) takes the reverse path. We create a single entry in the route tables at ports 2-3 to direct traffic to port 7 and a single entry in the route tables at ports 4-5 to direct return traffic to port 6. At ports 6 and 7, we create default entries.




For port 2:

• RLI Window: Select port 2 => Configuration => Route Table.
• Port 2 Window: Select Edit => Add Route.
• Port 2 Window: Enter 192.168.1.0/24 in the prefix/mask field and 7 in the next hop field.
• RLI Window: Select File => Commit.

  This step can be postponed until all route tables have been specified.



For ports 3-5:
• Repeat the above procedure for ports 3-5, taking care to enter the correct next hop entry (7 for ports 2 and 3 and 6 for ports 4 and 5).

For ports 6 and 7: We use default routing.
• RLI Window:
  Select port 6 => Configuration => Route Table.
• Port 6 Window:
  Select Edit => Generate Local Default Routes.
• Repeat the above steps for port 7
.
• RLI Window: Select File => Commit.


3. Install an egress GM filter at port 7 to direct all packets to a single queue

We will put all packets arriving to the egress side of port 7 into queue 300. So, packets from both UDP flows will be serviced from queue 300 in FIFO order. Because packets do not arrive in an alternating manner even when sent at the same rate, the bandwidth of each flow will vary over time. If we do not install this GM filter, the packets would go into one or two of the 64 datagram queues based on a hash function over some of the bits in their IP headers. By sending all packets to one queue, we make it easier to observe the queue length but harder to guarantee bandwidth.


• RLI Window: Select Port 7 => Configuration => Egress Filters.
• Port 7 Egress Filter: Select Edit => Add General Match Filter .
• Enter:
  • 0.0.0.0/0 in both the src and dest address fields

    This CIDR address will match any IP address.

  • * into the protocol field

    Although we know that we will be sending UDP packets, we can use this same filter if we send TCP packets or ICMP (ping) packets.

  • 300 into the qid field
  • 50 into the priority field
  


4. Set the link bandwidth at port 7

• RLI Window:
  Select Port 7 => Configuration => Queue Table.
• Port 7 Queue Table:
  Enter 300 in the Link Bandwidth dialogue box.
  


5. Configure egress queue 300 at port 7

   Egress queues have are two parameters. The threshold is the size of the queue in bytes; i.e., any packet that can not fit into the queue is dropped. The quantum is the Deficit Round Robin (DRR) credit parameter which determines the queue's share of the output link capacity. Since all packets will be put into queue 300, this parameter is irrelevant at this point.


• Port 7 Queue Table:
  Select Edit => Add Egress Queue.
• Edit the new queue descriptor so that the queue id is 300 and the threshold is 150000 bytes.
• We will accept the default quantum.
• RLI Window:
  Select File => Save to File, and save the configuration to a file.

  You can use this configuration file as a baseline for variations to the first experiment. Later, you can add monitoring charts to this same configuration file or save them in separate monitoring files which can be loaded as needed.

• RLI Window:
  Select File => Commit.


6. Create the bandwidth chart

   We will now create three charts: 1) Bandwidth, 2) Queue Length, and 3) Packet Drops. This step shows you how to create the Bandwidth chart which will show the traffic bandwidth coming from the two sources and the traffic bandwidth going to the two receivers by monitoring the counters inside the ingress side of the ATM switch.


• RLI Window:
  Select Monitoring => Add Monitoring Display.
• Graph Title Dialogue Box:
  Enter VOQ Bandwidth in the Name box.

  The VOQ Bandwidth chart will appear.



• VOQ Bandwidth Chart:
  Select Parameter => Add Parameter and select Enter.

  This moves the monitoring parameter focus to this chart. Since there is only one chart, this step was actually unnecessary.



• RLI Window:
  Select Port 2 => Monitoring => Ingress => Bandwidth to OPP.

  The Add Parameter dialogue box will appear.

• Add Parameter Dialogue Box:
  Enter 7 in the to port box, and select Enter.

  We will accept the default 1 second polling rate (period). The VOQ Bandwidth chart will appear with one line showing no traffic and labeled IPP 2 BW to OPP 7. We will now change the label to the more meaningful name 1.2 to 1.7 indicating bandwidth from NSP 1, port 2 to NSP 1, port 7.



• VOQ Bandwidth Chart:
  Select the legend label with the cursor.

  A dialogue box will appear



• Legend Dialogue Box:
  Change the name to 1.2 to 1.7 and uncheck the include ATM header box.

  This dialogue box indicates that a rate (bandwidth) is being computed from reading the number of ATM cells going from port 2 to port 7 of the ATM switch every 1.0 second.



• Legend Dialogue Box:
  Select Change to apply the name change to the chart legend.


• VOQ Bandwidth Chart:
  Repeat the above steps to monitor the traffic from 1.3 to 1.7 (IPP 3 BW to OPP 7), 1.6 to 1.4 (IPP 6 BW to OPP 4) and 1.6 to 1.5 (IPP 6 BW to OPP 5).


• VOQ Bandwidth Chart:
  Select Save To File, enter the File Name, and select Save.

  This will save the monitoring configuration to a file separate from the topology file created earlier. This approach allows us to dynamically combine different monitoring options with the topology file saved earlier. Alternatively, you could have incorporated the monitoring configuration into the topology file by using the RLI Window and enterring File => Save .



7. Create the queue length chart

   We monitor the congestion (queueing) at egress port 7 by monitoring the length of queue 300.


• RLI Window:
  Select Monitoring => Add Monitoring Display.
• Graph Title Dialogue Box:
  Enter Queue Length in the Name box.

  The Queue Length chart will appear.

• Queue Length Chart:
  Select Parameter => Add Parameter and select Enter.
• RLI Window:
  Select Port 7 => Configuration => Egress => QLength
• Add Parameter Dialogue Box:
  Enter 300 into the QID box and select Enter

  The Queue Length chart will appear.



8. Create the packet drop chart

   We monitor the number of packets arriving to the FPX at the egress side of port 7 that are being dropped.


• RLI Window:
  Select Port 7 => Configuration => Egress => FPX Counters
• Add Parameter Dialogue Box:
  Enter into the Counter box and select Enter


• Drops Chart:
  Select the legend label with the cursor.

  The P7Counter66 dialogue box will appear.

• P7Counter66 Dialogue Box:
  Change the name to 1.7 Drops and check the rate box to switch from absolute value.

  This dialogue box indicates rate or absolute number of packets being dropped at the egress side of the FPX at port 7 every 1.0 second.



9. Start the two UDP receivers

   We need to start iperf UDP servers at n1p4 and n1p5 to act as receivers. These receivers will run as background proceses until we kill them. After starting the receivers, we can start the two iperf UDP clients to act as the traffic senders. We will use the urcvrs-1nsp shell script to start both receivers and the usndrs-1nsp shell script to start both senders. Furthermore, we will run these scripts on the onlusr host which means that they will remotely launch the receivers and senders.


• Enter ("onlusr> " is the prompt):

  onlusr> ssh onl.arl.wustl.edu
  onlusr> urcvrs-1nsp >& log &
  onlusr> cat log.txt
  




The UDP iperf servers start running and wait for UDP iperf traffic on the internal network interfaces n1p4 and n1p5 respectively. The output shows that both servers are listening on UDP port 5001.

• You can verify that iperf is running on the two hosts by enterring:

  onlusr> source ~onl/.topology        # use .topology.csh if running a c-shell
  onlusr> ssh $n1p4 ps -lC iperf 
  onlusr> ssh $n1p5 ps -lC iperf
  

10. Start the two UDP senders

    Now that the two receivers are running, we can start sending traffic from the two senders at n1p2 and n1p3. We will run the


• Launch the two UDP senders in the foreground:
  

  usndrs-1nsp
  

The sender script staggers the starting time of the two senders to better distinguish between the two flows. The VOQ Bandwidth chart shows that the second sender starts about 8 seconds after the first one has started. The chart also shows that both senders are transmitting at about 200 Mbps (1.2 to 1.7 and 1.3 to 1.7). The 1.6 to 1.4 and 1.7 to 1.5 plots show that the combined flow rate is about 300 Mbps which is the bottleneck output rate that we set in the Queue Table. The output rate from port 7 of both flows varies during the contention period since the interpacket times for each flow are not constant. There will be short time periods when one flow gets more bandwidth than the other. The Queue Length chart shows that during the period that both flows are active, the queue length does reach its capacity of 150,000 bytes.




• The UDP iperf clients start running on the two hosts $n1p2 and $n1p3. Their output appears (perhaps intermixed) in the same window since the two sender processes were launched by the same process (the shell reading the script).




The output shows that the both clients have connected to their respective servers (n1p2 to n1p4 and n1p3 to n1p5) respectively, and have sent UDP traffic for 20 seconds at 200 Mbps. The output also shows that both servers have reported their final statistics back to their respective client processes. The first server reported that it received packets at a rate of 159 Mbps and observed that 22% of the expected packets were never received. The second server reported that it received packets at a rate of 180 Mbps and observed that 11% of the expected packets were never received. Both flows were able to transmit at higher than 150 Mbps (half of the link capacity) because they could transmit at 200 Mbps for some portion of their 20-second transmission period when the other flow was not active.

11. Stop the UDP senders

    The sender shell script periodically starts new UDP iperf senders and will continue indefinitely unless the user kills the sender script.

• Enter ctrl-c to kill the sender script during an inactive sender period.

  The two UDP iperf receivers are still running on hosts $n1p4 and 4n1p5.

This part repeats the experiment but with the two flows going to different egress queues at port 7 (queues 300 and 301). The packet scheduler will divide the 300 Mbps link capacity at port 7 according to the quantum values in the Queue Table. In the simplest case where both flows get an equal share of the link capacity, both flows will get 150 Mbps during congestion periods but with less variation than when both flows were placed into the same queue.

Steps (In Brief)

1. Direct the two flows into separate queues at port 7
2. Configure egress queue 301 at port 7
3. Add queue 301 to the Queue Length chart
4. Start the two UDP senders
5. Stop all UDP senders and receivers

Steps (In Detail)

Here are the major steps in this example:

1. Direct the two flows into separate queues at port 7

   We will direct packets from n1p2 to queue 300 and those from n1p3 to queue 301.


• Modify the GM filter at egress port 7 to direct packets from n1p2 to queue 300
• RLI Window:
  Select Port 7 => Configuration => Egress Filters.
• Port 7 Egress Filter:
  Select the src address field of the existing GM filter (Note: double click the box) and change the value to 192.168.1.48/28.

  This CIDR address will match the IP address of n1p2.

• Add a second GM filter at egress port 7 to direct packets from n1p3 to queue 301
• Port 7 Egress Filter: Select Edit => Add General Match Filter .
• Enter:
  • 192.168.1.64/28 in the src address field

    This CIDR address will match the IP address of n1p3.

  • * into the protocol field
  • 301 into the qid field
  • 50 into the priority field


2. Configure egress queue 301 at port 7

   We configure queue 301 to have the same properties as queue 300: a threshold of 150000 bytes and a quantum of 2,048.


• Port 7 Window:
  Select Table => Queue Table.
• Port 7 Queue Table:
  Select Edit => Add Egress Queue.
• Edit the new queue descriptor so that the queue id is 301 and the threshold is 150000 bytes.
• We will accept the default quantum.


3. Add queue 301 to the Queue Length chart
• Queue Length Chart:
  Select Parameter => Add Parameter and select Enter.
• RLI Window:
  Select Port 7 => Configuration => Egress => QLength
• Add Parameter Dialogue Box:
  Enter 301 into the QID box and select Enter

  The P7Q301 plot will appear in the Queue Length chart.

4. Start the two UDP senders

• Launch the two UDP senders in the foreground:
  

  usndrs-1nsp
  



• VOQ Bandwidth

The VOQ Bandwidth chart shows that the 1.6 to 1.4 and 1.7 to 1.5 plots have, as before, a combined flow rate of about 300 Mbps. But now, the flow rates are identical with no variation over time during the congestion period. The Queue Length chart shows that during the period that both flows are active, the queue length of both queues reaches their capacity of 150,000 bytes.



5. Stop all UDP senders and receivers

   In prepartion for repeating the experiment with TCP instead of UDP traffic, we need to terminate all of our iperf processes. The following sequence of commands can be put into a shell script and run from onlusr:

	onlusr> ssh $n1p2 pkill -n iperf
	onlusr> ssh $n1p3 pkill -n iperf
	onlusr> ssh $n1p4 pkill -n iperf
	onlusr> ssh $n1p5 pkill -n iperf

The above pkill commands will kill all of your most recent processes named iperf.

We can use the same setup for TCP traffic. Our setup will use the same forward path and use the reverse path for ACK packets sent by the receivers; i.e., ACK packets will go to port 6, then out the link back into port 7, and finally to the senders. The three charts will look different than those for UDP flows because of TCP's slow-start and congestion avoidance phases.

Steps (In Brief)

1. Start the two TCP receivers
2. Start the two TCP senders
3. Stop the two TCP senders

Steps (In Detail)

Here are the major steps in this example:

1. Start the two TCP receivers in the background

onlusr> trcvrs-1nsp >& log &
onlusr> cat log.txt

2. Start the two TCP senders

tsndrs-1nsp


• VOQ Bandwidth

The VOQ Bandwidth chart shows the flows jockeying back and forth for bandwidth as they go through their TCP congestion avoidance algorithm: as one flow backs off, the other flow increases its transmission rate. But on average, each flow gets about 150 Mbps during the congestion period.



• Queue Length

The Queue Length chart can be divided into two types of periods: no congestion and congestion. In the middle is the congestion period when both flows are contending for bandwidth. On either side of the congestion period, there is only one flow. So, that flow will consume the entire link capacity at port 7 and backoff only enough to keep the queue from overflowing. When packets begin arriving from n1p3 (the second flow), packets from the first flow will get dropped since queue 300 will only get half of its original transmission rate and the queue is already full. During this period, flow 2 will be increasing its transmission rate thus builing up the length of its queue.



• Packet Drops

The packet drop chart shows the initial spike of drops due to the first flow ramping up to the link rate during its slow-start phase and then backing off when it encounters packet drops. Then, when the second flow starts, it too ramps up rapidly but will detect packet drops earlier because its link share is 150 Mbps instead of 300 Mbps. Although the Queue Length chart doesn't show the queues hitting the limit of 150,000 bytes during this middle period, it would if the queue length counters were sampled more frequently.



• Iperf Client Report

  The client report shows that both TCP flows received about 200 Mbps of bandwidth during the 20 second interval (206 Mbps and 195 Mbps).


• Stop the two TCP senders
  • Enter ctrl-c to kill the sender script during an inactive sender period.

    The two TCP iperf receivers are still running on hosts $n1p4 and 4n1p5.

We delay ACK packets by 50 milliseconds by installing a delay plugin at egress port 6. Here are a couple of things to keep in mind:

• Plugins can not handle more than about 200 Mbps before they collapse because the SPC that implements the plugin environment is just an embedded 500 MHz Pentium III processor.
• You can not create a plugin until after the NSP has been committed.
• The list of available predefined plugins is not shown until after the NSP has been committed.

Steps (In Brief)

1. Install a delay plugin at egress port 6
2. Install an egress GM filter to direct packets to the delay plugin at port 6
3. Start the two TCP senders
4. Stop all of the TCP senders and receivers

Steps (In Detail)

1. Install a delay plugin at egress port 6

• RLI Window:
  Select Port 6 => Configuration => Plugin Table.

  The plugin table for port 6 will appear.



• Port 6 Plugins Table
  Select Edit => Add Instance => pdelay:600

  A partial plugin description for the pdelay plugin will appear.

• Port 6 Plugins Table:
  Enter 8 in the linked to spc qid box.

  This number will be used as a tag to bind to a GM filter. An acceptable number must fall into the range 8-127 because these are the IDs associated with the queues going to the SPC.



2. Install an egress GM filter to direct packets to the delay plugin at port 6

• Port 6 Egress Filters: Select Edit => Add GM Filter.
• Modify the spc qid field to 8 and the priority field to 50.

  The spc qid field value should be the same as value in the linked to spc qid field of the Plugins dialogue box. The spc qid field indicates which queue to place a matching packet. Packets in any queue with number in the range 8-127 are sent to the SPC.
  Alternatively, if we had enterred plugin instead of 8, the RLI will automatically provide a QID starting at 8. Note that the qid field is automatically computed by the RLI and is always 128 more in value than the valuein the spc quid field.

• RLI Window: Select File => Commit.

  After the commit finishes, the instance field in the plugin dialogue box will contain an instance a non-negative integer (starting from 0) indicating the plugin instance number.

3. Start the two TCP senders

tsndrs-1nsp

4. Stop all of the TCP senders and receivers

	onlusr> ssh $n1p2 pkill -n iperf
	onlusr> ssh $n1p3 pkill -n iperf
	onlusr> ssh $n1p4 pkill -n iperf
	onlusr> ssh $n1p5 pkill -n iperf

Concept Review

Debuging

• Enable debugging
• Display a debug message
• CAVEAT: Avoid SPC saturation

Packet Forwarding (XXX_handle_packet)

• Is called after the arrival of a pkt for the plugin instance
• All interrupts except a hard clock interrupt are disabled until you return from XXX_handle_packe.
• instance ptr (struct rp_instance *this)
• incoming pkt buffers (void *bufferList)

Queues

• xxx
• xxx
• xxx
• xxx
• xxx

Callbacks

• xxx
• xxx
• xxx
• xxx
• xxx
• xxx

Initialization

• xxx
• xxx
• xxx
• xxx
• xxx
• xxx

Control Messages

• xxx
• xxx
• xxx
• xxx
• xxx
• xxx

Miscellaneous

• If plugin instance ptr is NULL ==> object doesn't exist
• When writing a plugin that registers a callback function, ensure mutual exclusion within both the XXX_handle_packet and XXX_handle_callback functions by disabling all interrupts after gaining control (e.g., after enterring XXX_handle_packet and XXX_handle_callback) and enabling all interrupts before returning control (e.g., before returning from XXX_handle_packet and XXX_handle_callback).
• xxx
• xxx
• xxx
• xxx

Code Snippets

All .c files below are located in the appropriate subdirectory of ~onl/stdPlugins/. For example, COUNTER.c is located in ~onl/stdPlugins/COUNTER-200/COUNTER.c; The subdirectory name is formed from the prefix COUNTER and plugin number 200.

• XXX_handle_packet
• pkt buffers
• Ref: Tutorial => Summary Information => Summary of SPC Macros and Functions#XXX)
• Useful functions: msr_firstBuffer(bufList), msr_nextBuffer(curBuf), msr_removeBuffer(bufList, curBuf), msr_freeBuffer(curBuf)
• See ~onl/wu_arl/msr/rp/plugins/include/plugin_utils.h for definitions of buffer functions
• To forward an unmodified pkt
• Is default behavior if you don't do anything to pkt buffer list
• Pkt buffer should not be modified and be on pkt buffer list
• Example: COUNTER.c
• To drop a pkt
• Remove pkt from incoming pkt buffer list (msr_removeBuffer)
• Free pkt buffer (msr_free_buffer)
• Example: stats.c
• To duplicate a pkt
• Allocate space for a new pkt buffer (shim and IP pkt) (PLUGIN_IPBUF_ALLOC_FCT)
• Copy the shim and IP pkt to the new pkt buffer
• Forward the pkt by inserting buffer onto pkt list ( msr_addBuffer )
• Example: multicast.c
• To modify the payload of an IP pkt (but not its hdr)
  • Remember: The checksum field in the IP header covers only the header, not the payload. So, you can change the payload contents without having to change the IP header.
  • Usually the transport layer (e.g., TCP, UDP) header has a checksum field that covers the contents of the transport layer header and and payload. So, a new transport layer checksum must be computed if either its header or payload has been changed. ( calc_cksums , assign_tcpCksums , assign_udpCksums )
  • Example: stringSum.c
  • See ~onl/wu_arl/msr/rp/plugins/include/ipnet.h for definitions of checksum functions
• To modify an IP pkt hdr
• Remember: Pkt header fields will be in NBO.
• Locate the IP pkt hdr field(s) ( msr_pkt_iph , msr_iplen , msr_iphlen , msr_ipid , msr_ipttl , msr_ipdaddr , msr_ipsaddr , msr_ipproto , msr_ipdport , msr_ipsport )
• Convert the field to HBO (if necessary) ( ntohs , ntohl )
• Modify the field(s)
• Convert the field to NBO (if necessary) ( htons , htonl )
• Compute the IP hdr checksum ( wunet_cksum16 )
• Example: dumphdr.c, syn_demo.c
• See ~onl/wu_arl/msr/usr/src/sys/msr_ip.h for definitions of IP hdr functions
• To change the output port destination of a pkt
• Remember: This can only be done on the ingress side (where the VOQs are located).
• Change the OVIN (Output Virtual INterface) field in the shim (msr_shim_set_oving()).
• Example: multicast.c

Tutorial >> Examples	TOC