The ONL Tutorial

Tutorial >> Summary Information

TOC

ONL Linux 2.4 TCP

Linux 2.4 TCP is NOT Reno (nor Vegas, nor Tahoe) and can not be made to be exactly like these other TCP flavors without substantial kernel mods. There are three primary references:

• Linux 2.4 TCP is described in the paper Congestion Control in Linux TCP by Sarolahti and Kuznetsov;
• Tuning Linux 2.4 TCP is described in the Web page Enabling High Performance Data Transfers; and
• The TCP man page tcp(7).

The maximum socket buffer sizes have been set to 40 MB for both reading and writing, and the maximum per connection buffer sizes to 20 MB for both reading and writing. These values can be verified with:

	cat /proc/sys/net/core/[rw]mem_max	# max recv/send windows
	cat /proc/sys/net/ipv4/tcp_[rw]mem	# max TCP recv/send buffers

You can inspect the TCP tuning parameters either by examining the files /proc/sys/net/ipv4/tcp* using the cat or more commands or by the sysctl command. For example, to display all TCP parameters, try one of these two commands:

	more /proc/sys/net/ipv4/tcp*
	sysctl -a | grep tcp

All standard advanced TCP features are ON by default in the ONL testbed. Try one these commands:

	cat /proc/sys/net/ipv4/tcp_{timestamps,window_scaling,sack}
	sysctl net.ipv4.tcp_{timestamps,window_scaling,sack}

• The timestamp option is ON
  The sender places a timestamp in every TCP segment, and the receiver reflects the value in the ACK allowing the sender to calculate an RTT for each received ACK. This is important when you have large window sizes (large RTT) to get an accurate RTT estimate.
• The window scaling option is ON
  The 16-bit window size field in the TCP header is scaled by the window scaling factor. The scaling factor is transmitted during connection setup in the SYN segment, and takes effect only if both ends have window scaling turned ON. The value (between 0 and 14) indicates the amount that the window size field should be shifted to the left; i.e., the amount of doubling. This is important in large delay networks where the window size needs to be much larger than (the bandwidth-delay product is much larger than) 65,535 bytes.
• The SACK option is ON
  Selective acknowledgments (SACKs) (RFC 2018) are aimed at handling multiple packet drops within a window by having the receiver send back information about sequence holes in the receiver's receive buffer.

	sudo /usr/local/bin/net/timestamps-off
	sudo /usr/local/bin/net/sack-off

Very Long Delay Paths

If you are planning to experiment with a long-delay path, you should look at Yee-Ting Li's work. In short, there are low-level buffers that may be sized too small when you have long, fat pipes (delays of hundreds of milliseconds and Gbps rate). Unfortunately, Linux will silently drop packets when these buffers are full leaving no indication that you have lost packets at the endhost. But these parameters may have little utility with the delay plugin since it limits traffic to 200 Mbps. The following two commands allow you to increase the size of the receive and send buffers from their defaults.

• sudo /usr/local/bin/net/net-max-backlog
  This sets the maximum number of packets that can be queued at the receiver to 3,000 packets. The default is 300 packts. This is typically useful when using long fat pipes that have delays in the hundreds of milliseconds and Gbps transmission rate.
• sudo /usr/local/bin/net/ifconfig-txqueuelen
  This sets the maximum number of packets that can be queued at the sender. The default is 100 packts, but calling this script with no arguments sets it to 1,000 packets. This is typically useful when using long fat pipes that have delays in the hundreds of milliseconds and Gbps transmission rate. The script can also be called with one argument (the number of packet buffers) that is chosen from 100, 200, ... , 1000, 1500, 2000, 3000, ... , 10000.

Conformance to IETF

Readers should consult the Sarolahti and Kuznetsov paper for a detailed discussion of IETF conformance. This section summarizes what we consider to be the important differences for bulk transfer experiments in the ONL testbed. The table below is a reproduction of the conformance table in the Sarolahti and Kuznetsov paper.

Perhaps the most noticeable Linux 2.4 TCP feature is its retentive destination caching in which the ssthresh and RTO estimation parameters are cached for each destination. This means that once traffic has been sent to some destination D, the initial ssthresh value for the next TCP connection to D will be the same as the one at the end of the preceding connection to D. This is disturbing when you decide to change the packet delay to destination D and expect the initial ssthresh value to be infinite; i.e., stay in slow start until a packet drop occurs.

Below are comments on some of the features listed in the table:

• Control Block Sharing (RFC 2140)
  Linux's retentive destination caching is similar to RFC 2140 where connections to the same destination share the same control block parameters (e.g., ssthresh). The command

  	sudo /usr/local/bin/net/tcp-route-flush
  

  flushes the cache and in effect, makes ssthresh infinite. Unfortunately, it needs to be run before EACH new TCP connection.
• New Reno (RFC 2582)
  Linux fast recovery does not follow the New Reno fast recovery algorithm. First, the sender dynamically adjusts the threshold for triggering fast retransmit based on packet reordering so that fast retransmit may not be triggered by the third duplicate ACK. Second, the sender holds the window size during fast recovery instead of artificially adjusting it. These differences may be insignificant.
• Congestion Control (RFC 2581)
  Linux uses rate-halving (originally suggested by Hoe) rather than immediately halving the congestion window cwnd. In rate-halving, the sender decreases cwnd gradually, one segment for every two ACKs, until it reaches its cwnd/2 target. There is no parameter for modifying this behavior.
• RTO (RFC 2988)
  Linux follows the basic ideas in RFC 2988 for estimating the RTO but differs in two respects. First, the formula used for the RTO attempts to handle the case when there is a sudden drop in the RTT and when the RTT variance has dropped to a low value when the window size is large. Second, the minimum RTO is set to 200 msec instead of the 1000 msec prescribed in RFC 2988.

Tutorial >> Summary Information	TOC