The ONL NPR Tutorial

NPR Tutorial >> Examples

TOC

Filters, Queues and Bandwidth

Errata

• In some bandwidth charts below, the Y-axis label shows bytes/sec when the units were changed to Mb/s in the chart label dialogue box. This is due to a minor bug in the chart label dialogue box.

  Workaround: Using View => Change Y-axis Label in the monitoring window, sets the units for all of the plots in the chart.

Contents

• Example A: Two UDP Flows Share A Single Queue At Port 1.4
• Example B: Two UDP Flows Use Separate Queues At Port 1.4
• Common Problems/Mistakes
• Test Your Understanding

Introduction

We demonstrate the interaction of two 200-Mbps UDP flows going through a 300 Mbps bottleneck link (port 4 of the left NPR). The flows are created by the iperf utility. First, we show how to direct both flows through a single, reserved queue. Then, we show how to direct each flow to their own reserved queue.

The new concepts demonstrated by this example include:

• Using filters to direct packets to specific queues
• Setting the port rate (output link capacity) at a port
• Setting the threshold (capacity) and the quantum (bandwidth share) of an output queue
• Using the iperf command to generate UDP traffic
• Running shell scripts from the onlusr host to execute iperf on hosts assigned to your experiment
• Monitoring packets dropped by the Queue Manager

Preparation

In this example, two shell scripts are used to start traffic senders and receivers. You can get these by copying them from the directory /users/onl/export/NPR-Examples/ (after SSHing to the ONL user host onl.arl.wustl.edu) into the directory on the ONL user host where you store your executables and shell scripts (e.g., ~/bin/).

Although you can open up four windows, and SSH into the four hosts to run the iperf command, these scripts allow you to start the commands on the nodes from the ONL user host onl.arl.wustl.edu.

This example assumes that you have already created the SSH tunnel required by the RLI and started the RLI on your client host. It also assumes that you have a basic understanding of Unix commands. The example has two parts:

1. Two UDP flows share a single queue at port 1.4

   Packets from UDP flows contend for the bottleneck output link in a FIFO manner.

2. Two UDP flows use separate queues at port 1.4

   Packets from two UDP flows are placed in separate queues where each flow gets their share of the output link capacity. The user can specify the shares through the Queue Table at port 1.4. Separating the flows also smooths out the traffic bandwidth obtained by each UDP flow.

Steps (In Brief)

The main steps in Example A are:

1. Commit a dumbbell configuration
2. Install filters at ports 1.2, and 1.3
3. Set the link capacity at port 1.4
4. Configure queue 64 at port 1.4
5. Create the bandwidth chart
6. Create the queue length chart
7. Create the packet drop chart
8. Start the two UDP receivers
9. Start the two UDP senders
10. Terminate the UDP receivers

Steps (In Detail)

1. Commit a dumbbell configuration

   Create the dumbbell topology shown above with default route tables. The RLI example given earlier shows the steps in detail.


• Select Topology => Add NPR twice to show two NPRs and move them to the center of the RLI window with NPR.1 on the left.
• Use the spin handles to rotate each NPR icon into place.
• Select Topology => Add Host four times to get four hosts and drag each into position using the left mouse.
• Select Topology => Add Link and drag from the host icon to the center of port 3. Repeat for the other three host-NPR links and the NPR-NPR link from port 1.4 to 2.1.
• Select Topology => Generate Default Routes
• Select File => Commit.


2. Install filters at ports 1.2, and 1.3 to direct all UDP packets to a output single queue

Put all UDP packets arriving to either port 1.2 or 1.3 into queue 64, a reserved flow queue. The packets in queue 64 will be serviced in FIFO order. Because packets do not arrive in an alternating manner even when sent at the same rate, the bandwidth of each flow will vary over time. If we do not install this filter, the packets would go into one or two of the 64 datagram queues (numbered from 0 to 63) based on a hash function over some of the bits in their IP headers.
Remember: Queues 0-63 are datagram queues, and queues 64-8192 are reserved queues. By default, packets are placed into a datagram queue determined by a stochastic fair queueing algorithm (a hash function applied over IP header fields). The only thing special about queue 64 is that it is a reserved queue.


• RLI Window: Select Port 1.3 => Configuration => Filter Table.

  The Tables window for port 1.3 will appear with an empty filter table.



• Filter Table Window: Select Edit => Add Filter.

  A default filter table window will appear.



• Add Filter Window: Add or change the fields shown to the right.
  • protocol: Select udp

  You may choose to select * instead if you want to match any protocol.

  • port/plugin selection: Select port only
  • output ports: Enter 4
  • qid: Enter 64
  • Select Add



All UDP packets coming into port 1.3 will be placed in queue 64.
Note 1: The header of any IP packet will match the four source and destination fields since they specify don't care values.
Note 2: The priority of 50 gives the filter a higher priority than any matching route table entry which have a priority of 60.
Note 3: The other fields are not used.

• Install the same filter at port 1.2

  Repeat the above steps after opening up the Add Filter window at port 1.2.

3. Set the (outgoing) link capacity at port 1.4 to 300 Mbps

   Watch Out!!! This does NOT limit the incoming traffic rate, only the outgoing rate.


• RLI Window:
  Select Port 1.4 => Configuration => Queue Table.
• Port 1.4 Queue Table:
  Enter 300 in the Port Rate dialogue box.
  


4. Configure queue 64 at port 1.4

   Output queues have two parameters. The threshold is the size of the queue in bytes; i.e., any packet that can not fit into the queue is dropped. The quantum is the Deficit Round Robin (DRR) credit parameter which determines the queue's share of the output link capacity. Since all UDP packets will be put into queue 64, this parameter is irrelevant at this point.


• Port 1.4 Queue Table:
  Select Edit => Add Queue.
• Edit the new queue descriptor so that the queue id is 64 and the threshold is 1500000 bytes.
• Set the quantum to 1500.
• RLI Window:
  Select File => Commit.
• RLI Window:
  Select File => Save to File, and save the configuration to a file.

You can use this configuration file as a baseline for variations to the first experiment. Later, you can add monitoring charts to this same configuration file or save them in separate monitoring files which can be loaded as needed.



5. Create the bandwidth chart

   Now, create three charts: 1) Bandwidth, 2) Queue Length, and 3) Packet Drops. This step shows you how to create the Bandwidth chart which will show the traffic bandwidth coming from the two sources and the traffic bandwidth going to the two receivers by monitoring the byte counters inside NPR.


• RLI Window:
  Select Monitoring => Add Monitoring Display.
• Graph Title Dialogue Box:
  Enter bandwidth in the Name box.

  The bandwidth chart will appear.



• Bandwidth Chart:
  Select Parameter => Add Parameter.

  This moves the monitoring parameter focus to this chart. Since there is only one chart, this step was actually unnecessary.



• RLI Window:
  Select Port 1.3 => Monitoring => RXBYTE.

  The Add Parameter dialogue box will appear.



• Add Parameter Dialogue Box:
  Check the rate box, and select Enter.

  Accept the default 1 second polling rate (period). The bandwidth chart will appear with one line showing no traffic and labeled RXBYTE. Now change the label to the more meaningful name 1.3 indicating bandwidth at port 1.3 (perhaps RX 1.3 might be even clearer).



• RXBYTE Dialogue Box:
  Change the name to 1.3; select Mb/s as the units; and select Change (hidden beind the menu) to finish the dialogue.


• Repeat the above steps for the following monitoring points:
  • 1.2 RXBYTE (into 1.2),
  • 2.1 RXBYTE (into 2.1 from 1.4),
  • 2.2 TXBYTE (out 2.2), and
  • 2.3 TXBYTE (out 2.3).
  

  The only difference is that we will make the 2.1 line dashed instead of solid.



6. Create the queue length chart

   Monitor congestion at the bottleneck port by monitoring the length of queue 64 at port 1.4.


• RLI Window:
  Select Monitoring => Add Monitoring Display.
• Graph Title Dialogue Box:
  Enter queue length in the Name box.

  The queue length chart will appear.

• Queue Length Chart:
  Select Parameter => Add Parameter, and select Enter.
• RLI Window:
  Select Port 1.4 => Monitoring => ReadQLength

  The Add Parameter dialogue box will appear.



• Add Parameter Dialogue Box:
  Enter 64 into the queue_id box, and select Enter

  The Queue Length chart will appear with the line labeled as ReadQLength. Accept the 1-second polling rate, and monitor the absolute value of the queue length counter.



• Change the ReadQLength label to 1.4 queue 64, and the units to MB.


7. Create the packet drop chart

   Create another chart to display the number of packets dropped by the Queue Manager at port 1.4. This count is kept in register 31 (see below and Summary Information => Counters).


• RLI Window:
  Select Monitoring => Add Monitoring Display.
• Graph Title Dialogue Box:
  Enter QM Drops in the Name box.

  The QM Drops chart will appear.

• QM Drops Chart:
  Select Parameter => Add Parameter, and select Enter.
• RLI Window:
  Select NPR.1 => Monitoring => ReadRegisterPacket
  

The Add Parameter dialogue box will appear.
What's Happening? Each NPR has a set of numbered register pairs that count in bytes (ReadRegisterByte) and in packets (ReadRegisterPacket). These register numbers are listed in the tutorial page Summary Information => Counters.


• Add Parameter Window:
  Enter 31 in the index window, and select Enter.



The QM Drop chart appears with the line labeled as ReadRegisterPacket.
As indicated in the tutorial page Summary Information => Counters, counter 31 contains the number of packets dropped by the Queue Manager.


• QM Drops Window:

  We will change the name of the line from ReadRegisterPacket to QM Drops.

• Drops Chart:
  Open the ReadRegisterPacket dialogue box by clicking on the ReadRegisterPacket legend label.
• ReadRegisterPacket Dialogue Box:
  Change the name from ReadRegisterPacket to QM Drops and check the rate box to switch from absolute value.



This dialogue box indicates rate or absolute number of packets being dropped by the Queue Manager. (We could have selected rate in the Add Parameter dialogue box earlier.)

• RLI Window: File => Save As ...

  Save the configuration to a file, naming it udp.exp. This configuration file will be used in other examples.

8. Start the two UDP receivers

   Start iperf UDP servers on n2p2 and n2p3 to act as receivers by running the urcvrs-2npr shell script from the onlusr host. These receivers will run as background proceses until we kill them. The next step will show you how to start two iperf UDP clients on n1p2 and n1p3 to act as the traffic senders.

   
   • To run the urcvrs-2npr shell script, SSH to onl.arl.wustl.edu (not shown), and enter the command urcvrs-2npr.
   
   

   The script will start an iperf server process on each of the n2p2 and n2p3 hosts. The output from both processes will appear intermixed on your screen. Since the server reports are also displayed by the iperf clients when the clients are done sending, this output can be ignored except to check that a client has successfully connected to its server or if the server fails to successfully transmit its report to the client.
   Note: The UDP iperf servers start running and wait for UDP iperf traffic on the internal network interfaces. The output shows that both servers are listening on UDP port 5001.
   What's Happening? See the tutorial page Generating Traffic With Iperf for more details on shell scripts for generating traffic using iperf. The urcvrs-2npr script looks like this (with line numbers added):

   1   #!/bin/sh
   2   source /users/onl/.topology     # get defs of external interfaces
   3   for host in $n2p2 $n2p3; do     # start udp receivers
   4       ssh $host /usr/local/bin/iperf -s -u -w 1m &
   5   done
   

   • Line 1: A comment that says that /bin/sh should be used to interpret the shell script
   • Line 2: Define the environment variables associated with the control network interfaces. $n1p3 refers to the control network interface while n1p3 refers to the data (internal) network interface of the same host.
   • Lines 3-5: Line 4 is executed with $host replaced by the value $n2p2 in the first iteration and then by the value of $n2p3 in the second iteration; i.e., two iperf processes are started.
   • Line 4: iperf is executed as a background process in server mode (-s) for UDP packets (-u). In each case, a 1 MB (-w 1m) socket buffer is used.

   • You can verify that iperf is running on the two hosts by enterring:

     onlusr> source ~onl/.topology      # use .topology.csh if running a c-shell
     onlusr> ssh $n2p2 ps -lC iperf     # "ssh $n2p2 pgrep -l iperf" for less output
     onlusr> ssh $n2p3 ps -lC iperf
     

9. Start the two UDP senders

   Now that the two receivers are running, start sending traffic from the two senders at n1p2 and n1p3 using the usndrs-2npr shell script.

• In another command window, launch iperf UDP senders on hosts n1p2 and n1p3 by running the usndrs-2npr in the foreground:
  

  usndrs-2npr
  

• If the two iperf clients have successfully connected to their servers, you should see an indication in the receiver window. The last two lines in the figure below shows two connections:
  • server (192.168.2.48, port 5001) connected to client (192.168.1.48, port 32768), and
  • server (192.168.2.64, port 5001) connected to client (192.168.1.64, port 32768).

• The script staggers the starting time of the two senders to make it easiser to see the effect of the second flow on the traffic bandwidth.

The bandwidth chart shows that the second sender starts about 8 seconds after the first one has started. The chart also shows that both senders are transmitting at about 200 Mbps (1.2 and 1.3) when active. The 2.1 plot (green) shows that port 2.1 is receiving about 300 Mbps (the bottleneck output rate) when both senders are active. The 2.2 and 2.3 plots show the bandwidths transmitted to the two receivers and indicate that the sum of the two flow rates is 300 Mbps although it is possible for one flow to get more than 150 Mbps during the contention period. This unequal sharing of the link capacity is possible because the interpacket times for each flow are not constant.




The queue length chart shows that queue 64 has packets during the congestion period [618, 630] when both flows are active. Furthermore, queue 64 peaks at about 1.5 MB, the queue threshold.




During this same congestion period, the Queue Manager is dropping packets. Furthermore, about 9,447 packets are dropped each second.




What's Happening? The usndrs-2npr script looks like this:

1   #!/bin/sh
2   source /users/onl/.topology    # get defs of external interfaces
3   ssh $n1p2 /usr/local/bin/iperf -c n2p2 -u -b 200m -t 20 &
4   sleep 8
5   ssh $n1p3 /usr/local/bin/iperf -c n2p3 -u -b 200m -t 20 &

• Lines 1-2: Are the same as in urcvrs-2npr.
• Line 3: Start an iperf process in the background (&) on host $n1p2. $n1p2 is the value of a shell variable and refers to the control network interface name of the host labeled n1p2 in the RLI (e.g., onl054). The iperf process will send UDP packets (-u) at 200 Mbps (-b 200m) for 20 seconds (-t 20).
• Line 4: Line 4 ensures that the second sender doesn't start until 8 seconds after the first one has started.
• Line 5: This is the same as Line 3 but starts iperf on host $n1p3 instead of $n1p2.



Normally, the output from both iperf will be intermixed. But the above figure shows the output from each process separated into its own file. Both senders sent at 203 Mbps for 20 seconds. But the server reports at the end of each session indicate that n2p2 only saw 181 Mbps with 11% of the packets dropped and n2p3 only saw 159 Mbps with 21% of the packets dropped.
Watch Out!!! The bandwidth reported by iperf excludes the bits in the IP header and UDP header. That is, 1470-byte IP datagrams are sent (20-byte IP header, 8-byte UDP header, 1442-byte UDP payload), but the bandwidth shown only includes the 1442 bytes in the UDP payload. So, the 203 Mbps reported by the senders is really 207 Mbps (= 203 x 1470/1442). Both servers received traffic at higher than 150 Mbps (half of the link capacity) because they could transmit at 200 Mbps for some portion of their 20-second transmission period when the other flow was not active.

10. Terminate the UDP receivers
• The sender processes will terminate on their remote hosts when they are done sending their 20 seconds of traffic.
• Terminate the receivers by sending them the kill command.

  You can do that remotely from onlusr or SSH to each host running iperf.

  Here is how to do it from onlusr:

onlusr> source /users/onl/.topology    # .topology.csh if using a c-shell
onlusr> ssh $n2p2 pkill iperf          # kill all iperf on $n2p2
onlusr> ssh $n2p3 pkill iperf          # kill all iperf on $n2p3

---

Example B: Two UDP Flows Use Separate Queues At Port 1.4

This part repeats the experiment in Example A but with the two flows going to different queues at port 1.4: traffic from n1p2 goes into queues 65 and traffic from n1p3 goes into queues 64. The packet scheduler will divide the 300 Mbps link capacity at port 1.4 according to the quantum values in the Queue Table. In the simplest case where both flows get an equal share of the link capacity, both flows will get 150 Mbps during congestion periods but with less variation than when both flows were placed into the same queue. The example also shows how to give the flow from n1p2 twice as much bandwidth (200 Mbps) as the one from n1p3 (100 Mbps).

Steps (In Brief)

1. Direct the two flows into separate queues at port 1.4
2. Configure queue 65 at port 1.4
3. Add queue 65 to the Queue Length chart
4. Start the UDP receivers and senders
5. Change the bandwidth shares at port 1.4
6. Terminate UDP receivers

Steps (In Detail)

The main steps in Example B are:

1. Direct the two flows into separate queues at port 1.4

   Direct packets from n1p2 to queue 65 and those from n1p3 to queue 64. Since we have already installed the filter at port 1.3, we only need to install one at port 1.2.


• Add a filter at port 1.2 to direct packets from n1p2 to queue 65

  This filter is identical to the one we installed at port 1.3 except that the qid field value is now 65 instead of 64.



2. Add and configure queue 65 at port 1.4

   Configure queue 65 to have the same properties as queue 64: a threshold of 1500000 bytes and a quantum of 1,500.


• Port 7 Window:
  Select Table => Queue Table.
• Port 7 Queue Table:
  Select Edit => Add Egress Queue.
• Edit the new queue descriptor so that the queue id is 301 and the threshold is 750000 bytes (half of 1500000).
• Accept the default quantum.
• Modify the threshold for queue 64 to 750000
• RLI Window: Select File => Commit.


3. Add queue 65 to the Queue Length chart

   The steps are identical to the ones for queue 64.

• Queue Length Chart:
  Select Parameter => Add Parameter, and select Enter.
• RLI Window:
  Select Port 1.4 => Monitoring => ReadQLength

  The Add Parameter dialogue box will appear.

• Add Parameter Dialogue Box:
  Enter 65 into the queue_id box, and select Enter

• Change the ReadQLength label to 1.4 queue 65; change the units to MB; select the Dashed Line checkbox; and select Change.


• RLI Window: File => Save As ...

  Save the configuration to a different file than in Example A. This configuration file will be used in other examples.

4. Start the UDP receivers and senders

   The procedure was described in Example A.


• Compare this bandwidth chart with the one in Example A. The 2.2 and 2.3 plots in the congestion interval [3,148, 3,160] are identical(about 150 Mbps); i.e., both flows now get an equal share of the 300 Mbps link capacity at port 1.4.


• Both queue 64 and 65 peak about about 0.75 MB, the queue thresholds.


• The total number of packets dropped by the Queue Manager remains about the same as in the experiment in Example A.


5. Change the bandwidth shares at port 1.4

   Change the bandwidth shares at port 1.4 so that the flow from n1p2 gets twice as much bandwidth (200 Mbps) as the one from n1p3 (100 Mbps).


• RLI Window: Select Port 1.4 => Queue Table.
• Port 1.4 Queue Table: Change the quantum value for qid 65 from 1500 to 3000.
• RLI Window: Enter File => Commit.
• If the UDP senders are no longer running, restart the senders.


• The bandwidth chart should now show that during the congestion period the 2.2 plot is getting 200 Mbps and the 2.3 plot is getting 100 Mbps (i.e., the ratio of the bandwidth to n2p2 through port 2.2 to the bandwidth to n2p3 through port 2.3 is 3000:1500 or 2:1).


6. Terminate UDP receivers

   The procedure was given in Example A.

---

Common Problems/Mistakes

Here are some common problems/mistakes:

• The port rate at port 2.1 has no effect

  Setting the port rate of port 2.1 to say 10 Mbps will show no effect in our examples because the port rate determines the link capacity of the outgoing traffic, not the input link rate. Since the only traffic going from port 2.1 to 1.4 are the few packets sent by the iperf servers to the clients, there is very little traffic returning from NPR 2.

• Iperf command not found

  Reason 1: You can run the individual iperf commands directly on the hosts that should be generating the traffic instead of the onlusr host. Often, users attempt to run iperf from onlusr, without first using SSH to login to the intended host. Since onlusr has no iperf executable, the command will not be found.
  Reason 2: The iperf command is installed at /usr/local/bin/iperf on all of the ONL hosts except onlusr. But the directory /usr/local/bin must be in your PATH environment variable in order to avoid having to give the full pathname /usr/local/bin/iperf.

• iperf Transmission Rate

  Iperf reports traffic bandwidth as seen at the application level. This bandwidth does not include the IP header and the transport-level header (UDP, TCP). The effect is that measured bandwidth will be higher. The amount will depend on the ratio (H+A)/A where H is the number of bytes in the network- and transport-headers and A is the number of bytes in the application data unit.

---

Test Your Understanding

Tipping Point Experiments



Start with the configuration for Example A where both UDP flows go to queue 64 at port 1.4. Repeat the experiment in Example A with the port rate at port 1.4 set to 412 Mbps.

• Do you get the charts shown below?
• Explain why the charts make sense?


• The 2.2 and 2.3 bandwidths are both approximately 203 Mbps.


• There are packet drops but only a small number (26).


• The length of queue 64 grows slowly (1.5 MB in about 12 seconds); i.e., at a rate of about 1 Mbps.



Bandwidth Sharing Experiments

Add hosts at ports 1.1 and 2.4 and modify the experiment in Example A so that the bandwidths through the bottleneck link at port 1.4 are in the ratio 1:2:3 for traffic from ports 1.1, 1.2, and 1.3 respectively.

Revised: Wed, Sep 17, 2008

---

NPR Tutorial >> Examples	TOC