The ONL NPR Tutorial

NPR Tutorial >> Writing A Plugin

TOC

Packet Processing Functions

Contents

• Dropping a Packet
• Reading an IP Header
• Reading a TCP/UDP Header
• Changing the Next Block and Plugin Tag
• Send meta-packet back for reclassification
• Set the plugin tag to (1 + TOS) and reclassify packet
• Send meta-packet to another plugin
• Modifying a Packet Body
• Modifying a Packet Header
• Monitoring Queue Length

There are predefined plugin code examples that demonstrate basic packet processing functions. Below is a table listing some of the functions and plugins that contain the code. The first plugin listed is the one that we cite as an example here. The ones enclosed in parentheses also demonstrate the feature but are not discussed here.

All of the packet processing functions are summarized in the page NPR Tutorial => Summary Information => Plugin Functions . In addition to the core packet processing functions (e.g., drop a packet, compute a checksum), the page also summarizes other useful functions such as string manipulation functions (e.g, strncmp() to compare two strings) and memory functions (e.g., memcpy() to copy memory).

These plugins use a variety of functions. These functions are documented in NPR Tutorial => Summary Information => Plugin Functions and summarized by category in the table below.

The files plugin_api.h, plugin_dl.h and plugin_helpers.h can be found at ~onl/npr/pluginFramework/ on the onlusr host. More advanced users like to see the coding details of lower-level functions or have a need to cannabalize code may be intersted in the source code located in the subdirectories of ~onl/npr/onl_router/.

Watch Out! Do not think that just because there is a stdlib.h that the same functions available in standard libraries on general-purpose OSes (e.g., glibc) are available to the plugin programmer. Check NPR Tutorial => Summary Information => Plugin Functions first.

Dropping a Packet

Some of the reasons you may want to drop a packet are:

• The plugin is acting as a passive monitor and is receiving a copy of the original packet (i.e., the packet matched an auxiliary filter);
• A policy specifies that certain packets be dropped; or
• The packet would be a danger to the network (e.g., the packet contains a virus).

Recall that handle_pkt() looks like this:

	void handle_pkt()
	{
	    dl_source_packet(dlFromBlock);
	    default_format_out_data(dlNextBlock);
	    handle_pkt_user();
	    dl_sink_packet(dlNextBlock);
	}

For example, the drop plugin looks like this where the two functions onl_api_set_out_to_DROP() and onl_api_set_out_to_QM() set dlNextBlock to the appropriate value:

	void handle_pkt_user()
	{
	    __declspec(local_mem) unsigned int drop = 0;
	    ... Do processing and set value of drop ...
	    if(drop == 1)	onl_api_set_out_to_DROP();
	    else 		onl_api_set_out_to_QM();
	}

The nstats plugin drops every packet it gets. So, it calls onl_api_set_out_to_DROP() as the last thing it does for every packet.

Plugin processing often depends on one or more values in an IP header. For example, nstats updates one of three counters based on the value in the protocol field. Or, we may want to drop to drop packets coming from an IP address that appears on a quarantine list. Or, we may want to display several fields in the IP header as part of a debug message.

The task of reading any field in the IP header is straightforward. It is only made non-trivial for two reasons:

• There is a buffer descriptor and a packet buffer associated with every unicast packet.
• The plugin gets only a meta-packet and not the packet itself.

But the address of both the packet descriptor and the packet buffer itself is computed from the buffer handle that is stored in the meta-packet. The two computations are:

	Let	H be the buffer handle
		B be the address of SRAM bank 2
	Then
		descriptor_address = H<<2 + B
		buffer_address = H<<8

Since the buffer handle is stored in the meta-packet sent to each plugin, we can easily compute the location of both the buffer descriptor in SRAM and the packet in DRAM. Furthermore, there are functions that do that for us along with functions for reading the IP header from DRAM to local memory. So, it is easy (below) to see how the nstats plugin determines the protocol of the packet below:

	void handle_pkt_user()  {
1	    __declspec(gp_reg) onl_api_ip_hdr ipv4_hdr;
2	    __declspec(gp_reg) buf_handle_t buf_handle;
3	    __declspec(gp_reg) onl_api_buf_desc bufDescr;
4	    unsigned int bufDescPtr;
5	    unsigned int ipv4HdrPtr;
6	    unsigned int dramBufferPtr;
7	
8	    onl_api_get_buf_handle(&buf_handle);
9	    bufDescPtr = onl_api_getBufferDescriptorPtr(buf_handle);
10	    onl_api_readBufferDescriptor(bufDescPtr, &bufDescr);
11	    dramBufferPtr = onl_api_getBufferPtr(buf_handle);
12	    ipv4HdrPtr = onl_api_getIpv4HdrPtr(dramBufferPtr, bufDescr.offset);
13	
14	    onl_api_readIpv4Hdr(ipv4HdrPtr, &ipv4_hdr);       
15	
16	    switch(ipv4_hdr.ip_proto) { ... Increment plugin counter ... }
17	
18	    onl_api_drop();
	}

Lines 8-12 compute the address (in DRAM) of the IPv4 header starting from the buffer handle that is in the meta-packet. From our earlier description of how to find the packet buffer address of a packet, finding the IPv4 header in DRAM is straightforward since it is at a fixed distance from the beginning of the buffer.

• Line 8: Returns the current packet's buffer handle in buf_handle by reading it from the meta-packet.
• Line 9: Computes the address of the packet buffer descriptor in SRAM given the buffer handle.
• Line 10: Reads the packet buffer descriptor into bufDescr (stored in general-purpose registers) given the address of the buffer descriptor in SRAM.
• Line 11: Computes the address of the packet buffer in DRAM given the buffer handle.
• Line 12: Computes the address of the IPv4 header in DRAM given the address of the DRAM buffer and the offset field in the buffer descriptor. For ONL, the offset is always 0x18E (398 decimal).
• Line 14: Reads the IPv4 header from DRAM to the plugin's general-purpose registers.
• Line 16: Increments one the three plugin counters based on the packet's protocol number.

Other fields in the IP header can be examined in a similar manner. You would use lines 8-14 to read the IP header into general-purpose registers where the field(s) of interest can be examined. The declaration of the IP header structure (onl_api_ip_hdr) can be found in ~onl/npr/pluginFramework/onl_api.h.

One final note is that the IXP stores integers in network byte order which means that the IXP understands the long and short integers as they come off of the network and the programmer does not have to use the byte ordering functions (e.g., ntohl) that are used by convention on endhosts.

Reading fields from a TCP or UDP header is not much different than reading fields from an IP header. The location of both a UDP or TCP header can be found by getting the IP header length L from the IP header and adding 4L to the beginning of the IP header (since L is the number of words in the IP header). Once you have the UDP or TCP header address, you read the header from DRAM into registers or local memory.

There are four functions associated with reading UDP and TCP headers:

• onl_api_getUdpHdrPtr(): Return the address of the UDP header given the address of the IP header and IP header length (number of words).
• onl_api_readUdpHdr(): Read the UDP header from DRAM given the address of the UDP header and the address of the header copy.
• onl_api_getTcpHdrPtr(): Return the address of the TCP header given the address of the IP header and IP header length (number of words).
• onl_api_readTcpHdr(): Read the TCP header from DRAM given the address of the TCP header and the address of the header copy.

Here is an example of how to read a UDP header:

	__declspec(gp_reg) onl_api_ip_hdr	ipv4_hdr;
	__declspec(gp_reg) onl_api_udp_hdr	udp_hdr;
	unsigned int				udpHdrPtr;
	... Set ipv4HdrPtr and ipv4_hdr ...
	udpHdrPtr = onl_api_getUdpHdrPtr(ipv4HdrPtr, ipv4_hdr.ip_hl);
	onl_api_readUdpHdr(udpHdrPtr, &udp_hdr);

The default destination for a meta-packet leaving a plugin is the Queue Manager. If you want to change the destination to one of the other blocks, you must form a different meta-packet and indicate which block should receive the meta-packet. These steps are done in handle_pkt_user().

The following example is taken from the TOStag plugin. The handle_pkt_user() routine reads the TOS field from the IP header, sets the plugin tag field in the meta-packet to (1 + TOS), and sends the meta-packet back to MUX which will in turn send it to PLC for reclassification.

1	__declspec(shared gp_reg) unsigned int lastTag;
2
3	void handle_pkt_user()  {
4	    __declspec(gp_reg) onl_api_ip_hdr ipv4_hdr;
5	    __declspec(gp_reg) buf_handle_t buf_handle;
6	    __declspec(gp_reg) onl_api_buf_desc bufDescriptor;
7	    __declspec(sram) unsigned int bufDescPtr;
8	    __declspec(sram) unsigned int ipv4HdrPtr;
9	    __declspec(sram) unsigned int dramBufferPtr;
10
11	    onl_api_plugin_cntr_inc(pluginId, 0);	// plugin cntr 0 
12
13	    onl_api_get_buf_handle( &buf_handle );	// get IP hdr addr
14	    bufDescPtr = onl_api_getBufferDescriptorPtr( buf_handle );
15	    onl_api_readBufferDescriptor( bufDescPtr, &bufDescriptor );
16	    dramBufferPtr = onl_api_getBufferPtr( buf_handle );
17	    ipv4HdrPtr = onl_api_getIpv4HdrPtr( dramBufferPtr,
18							bufDescriptor.offset );
19	    onl_api_readIpv4Hdr( ipv4HdrPtr, &ipv4_hdr );	// read IP hdr
20
21	    lastTag = ipv4_hdr.ip_tos + 1;
22	    helper_set_meta_default( MUX );
23	    helper_set_meta_mux_tag( lastTag );
24	}

The new lines of code are lines 21-23. Line 11 had been used in the mycount plugin, and lines 13-17 are standard lines used whenever we read the IP header to local memory.

• Line 21: Set the plugin tag lastTag to one more than the TOS field. We use a filter to match this plugin tag to identify packets with certain TOS values. This value is later used in line 23.
• Line 22: This call sets the fields in the ring_out meta-packet to default values for the MUX block and also sets dlNextBlock to MUX, a constant defined in ~onl/npr/pluginFramework/plugin_dl.h. Lines 21 and 22 make it possible to call dl_sink_packet( dlNextBlock ) in handle_pkt() in a meaningful way.
• Line 23: Changes the plugin tag field in the ring_out meta-packet to the desired value since its default value set in line 22 is 0.

This approach where handle_pkt_user() sets up the meta-packet requires changes to the handle_pkt() processing:

	void handle_pkt()
	{
1	    dl_source_packet( dlFromBlock );
2	    handle_pkt_user( );
3	    dl_sink_packet( dlNextBlock );
	}

We have eliminated the line that calls default_format_out_data() which sets up the outgoing meta-packet (ring_out) based on the incoming meta-packet (ring_in). This functionality has been replaced by the call to helper_set_meta_default() in handle_pkt_user().

The two functions helper_set_meta_default() and helper_set_meta_mux_tag() are functions in the plugin_helper.h file and not (as yet) part of the plugin API.

When modifying the contents of a packet (the packet body), you must locate and change the field of interest and update the transport layer (TCP, UDP) checksum. This process is complicated by the fact that the packet body is located in DRAM and often requires multiple reads to scan the entire packet. Below is the handle_pkt_user() routine for the stringSub routine that replaces occurrences of the string "hello" with the string "adieu" from a 64-byte UDP packet.

	void handle_pkt_user()  {
1	    __declspec(gp_reg) onl_api_ip_hdr ipv4_hdr;
2	    __declspec(gp_reg) buf_handle_t buf_handle;
3	    __declspec(gp_reg) onl_api_buf_desc bufDescriptor;
4	    __declspec(gp_reg) onl_api_udp_hdr udp_hdr;
5	    __declspec(local_mem) sixteenWordsOfData payload;
6	    __declspec(local_mem) unsigned int bufDescPtr;
7	    __declspec(local_mem) unsigned int ipv4HdrPtr;
8	    __declspec(local_mem) unsigned int dramBufferPtr;
9	    __declspec(local_mem) unsigned int udpHdrPtr;
10	    __declspec(local_mem) unsigned int payloadPtr;
11	    __declspec(local_mem) char* q;  
12	    __declspec(local_mem) int cnt = 0;
13	
14	    onl_api_get_buf_handle( &buf_handle );
15
16	    bufDescPtr = onl_api_getBufferDescriptorPtr( buf_handle );
17	    onl_api_readBufferDescriptor( bufDescPtr, &bufDescriptor );
18
19	    dramBufferPtr = onl_api_getBufferPtr( buf_handle );
20	    ipv4HdrPtr = onl_api_getIpv4HdrPtr( dramBufferPtr, bufDescriptor.offset );
21	    onl_api_readIpv4Hdr( ipv4HdrPtr, &ipv4_hdr );
22	
23	    udpHdrPtr = onl_api_getUdpHdrPtr( ipv4HdrPtr, ipv4_hdr.ip_hl );
24	    onl_api_readUdpHdr( udpHdrPtr, &udp_hdr );
25							// get payload from DRAM
26	    payloadPtr = onl_api_getUdpPacketPayloadPtr( udpHdrPtr );
27	    onl_api_ua_read_8W_dram( payloadPtr, (void *) &payload );
28	    onl_api_ua_read_8W_dram( payloadPtr+32, (void *) &(payload.i[8]) );
29							// do substitution
30	    q = (__declspec(local_mem ) char*) &( payload.i );
31	    while( cnt < 16*4 ) {
32		if( prefix(findStr,q) )		copy( replaceStr,q );
33		cnt++;
34		q = ( (__declspec(local_mem ) char*) &(payload.i)) + cnt;
35	    }
36							// new payload to DRAM
37	    onl_api_ua_write_8W_dram( payloadPtr, (void *) &payload);
38	    onl_api_ua_write_8W_dram( payloadPtr+32, (void *) &(payload.i[8]) );
39					// update udp hdr (with checksum)
40	    udp_hdr.uh_sum = onl_api_udp_cksum( &ipv4_hdr, &udp_hdr, udpHdrPtr );
41	    onl_api_writeUdpHdr( udpHdrPtr, &udp_hdr );
	}

Before discussing the code lines, recall from the section Reading an IP Header earlier that there is a direct correlation between a buffer handle H, the location B of the packet buffer, and the location D of the packet descriptor. That section showed that the location of the IP header, UDP header and UDP payload can be determined by starting with the location of the packet buffer. The functions in lines 14-24 above allow you to determine these locations and ultimately the location of the UDP payload in the packet buffer without remembering the implementation details.

• Lines 1-12: All of the variables in handle_pkt_user() are stored in general-purpose registers or local memory for fast access.
• Line 14: The function onl_api_get_buf_handle() returns the buffer handle buf_handle of the incoming packet. This information is actually stored in the current meta-packet in the plugin's input ring.
• Lines 16-17: Reads the 8-word (or 32-byte) packet descriptor into general-purpose registers. The beginning of the IP packet within the packet buffer is given by the offset field in the packet descriptor and is used in Line 20 to locate the beginning of the IP packet.
• Lines 19-21: Reads the IPv4 header into local memory so that we can locate the beginning of the UDP header.
• Lines 23-24: Reads the UDP header into local memory so that we can update its checksum later in Lines 40-41.
• Lines 26-38: Reads the 64-byte UDP payload in two 32-byte segments; replaces all occurrences of the five character sequence "hello" with the five character sequence "adieu"; and then writes the new payload back to SRAM in two 32-byte segments. Note that this process doesn't change the UDP payload length (nor the IP payload length). The two functions onl_api_ua_read_8W_dram() and onl_api_ua_write_8W_dram() do the reading from and writing to DRAM.
• Lines 40-41: Compute the new UDP checksum and write the new UDP header back to DRAM. Note that the checksum in the IP header doesn't need to be recomputed because it only covers the IP header which has not been changed by the substitution.

The key ideas here are that if you modify a field in an IP header, you will need to update the IP checksum in the packet buffer; and if you modify a field in an UDP/TCP header, you will need to update the UDP/TCP checksum in the packet buffer. If the checksum in the incoming UDP header is 0, then the UDP checksum should not be computed. But the UDP/TCP checksum includes a 12-byte pseudo-header that includes fields from the IP header (source and destination IP addresses and protocol);

	void handle_pkt_user()  {
1	    __declspec(gp_reg) onl_api_ip_hdr ipv4_hdr;
2	    __declspec(gp_reg) buf_handle_t buf_handle;
3	    __declspec(gp_reg) onl_api_buf_desc bufDescriptor;
4	    __declspec(gp_reg) onl_api_udp_hdr udp_hdr;
5	    __declspec(local_mem) sixteenWordsOfData payload;
6	    __declspec(local_mem) unsigned int bufDescPtr;
7	    __declspec(local_mem) unsigned int ipv4HdrPtr;
8	    __declspec(local_mem) unsigned int dramBufferPtr;
9	    __declspec(local_mem) unsigned int udpHdrPtr;
10	
11	    onl_api_get_buf_handle( &buf_handle );
12
13	    bufDescPtr = onl_api_getBufferDescriptorPtr( buf_handle );
14	    onl_api_readBufferDescriptor( bufDescPtr, &bufDescriptor );
15
16	    dramBufferPtr = onl_api_getBufferPtr( buf_handle );
17	    ipv4HdrPtr = onl_api_getIpv4HdrPtr( dramBufferPtr, bufDescriptor.offset );
18	    onl_api_readIpv4Hdr( ipv4HdrPtr, &ipv4_hdr );
19	
20	    udpHdrPtr = onl_api_getUdpHdrPtr( ipv4HdrPtr, ipv4_hdr.ip_hl );
21	    onl_api_readUdpHdr( udpHdrPtr, &udp_hdr );
22
23	    ... Modify one or more header fields ...
24					// update ip chksum, ip hdr, udp hdr
25	    ipv4_hdr.ip_sum = onl_api_ipv4Hdr_cksum16( &ipv4_hdr );
26	    onl_api_writeIpv4Hdr( ipv4HdrPtr, &ipv4_hdr );
27
28	    udp_hdr.uh_sum = onl_api_udp_cksum( &ipv4_hdr, &udp_hdr, udpHdrPtr );
29	    onl_api_writeUdpHdr( udpHdrPtr, &udp_hdr );
	}

The code above is almost identical to that in the section Modifying a Packet Body except for the lines starting at Line 23. The major difference is that Lines 25-26 assume that both the IP header checksum and the UDP header checksum need to be recomputed.

You may want to process a packet differently depending on the length of a queue or you may just want to read the parameters (e.g., quantum, threshold) associated with a queue. The function onl_api_getQueueParams() allows you to read the following parameters of a particular queue:

• length (in bytes)
• threshold
• quantum

The following example is taken from the priq plugin which implements strict priority queueing for three flow priorities. The plugin assumes that the user will install a filter that will send packets to the plugin with a qid field of 1, 2 or 3 indicating GOLD (high priority), SILVER (medium priority) or BRONZE (low priority) flows respectively. When a packet arrives to the priq plugin, the handle_pkt_user() routine examines the qid field in the meta-packet and takes one of three actions:

• GOLD: Immediately forward the packet to the Queue Manager for queue 4/64 (i.e., output port 4, QID 64).
• SILVER: Place the packet into its own queue 0 (SILVERQ).
• BRONZE: Place the packet into its own queue 1 (BRONZEQ).

	void callback() {
1	    __declspec( gp_reg ) onl_api_qparams	qparams;
2
3	    onl_api_getQueueParams( 41024, &qparams );	// 5*8192+64
4	    if( qparams.length == 0 ) {		// empty high-priority queue
5	    	... Forward highest priority packet to queue 4/64 ...
6	    }
7
8	    sleep( SLEEP_CYCLES );
	}

The code is straightforward.

• Line 1: The data type onl_api_qparams is defined in the plugin_api.h file to be a four-word structure in which the fourth word is unused and the first three contain the queue length (in bytes), the threshold (in bytes), and the quantum.
• Line 3: Note that each port has 8,192 queues with datagram queues numbered from 0 to 63 and reserved queues numbered from 64 to 8191. Queue 4/64 is represented by the internal queue number 41024 which is derived from the output port number and the QID using the formula (1+port)*8192 + QID. For the priq plugin, port = 4 and QID = 64.
• Lines 4-6: The plugin only forwards a packet from one of its internal queues if queue 4/64 is empty.

NPR Tutorial >> Writing A Plugin	TOC