The ONL NPR Tutorial

NPR Tutorial >> Writing A Plugin

TOC

Tour of a Simple Plugin

Contents

• Useful Files
• A Few Caveats
• General Structure of a Simple Plugin
• The mycount Plugin
• The Plugin API and Helper Functions
• Thread-Specific Global Variables
• Main
• Plugin Initialization
• Handling a Packet
• Handling a Control Message
• Some Comments on the Makefile
• What's Left to Learn?

The source code for all of the predefined plugins are in subdirectories of ~onl/npr/plugins/ in the ONL filesystem. For example, ~onl/npr/plugins/mycount/ is the directory for the mycount plugin, and ~onl/npr/plugins/mycount/mycount.c is the source file for the plugin. These files are important to the plugin developer because most developers start from the source code for a plugin that serves as the starting point for the code. All of the source code have the same base, and the null plugin is the required plugin code stripped of all functionality other than packet forwarding. Every plugin directory has a README file that describes the plugin and the degree to which the plugin has been tested. Most of these plugins are summarized in the table in the NPR Tutorial page " NPR Tutorial => Router Plugins => Predefined Plugins".

As discussed later, the files in ~onl/npr/ may also be worthwhile: the .h files in the subdirectory pluginFramework/ contain the functions that make up the plugin API. Also, reading the code may give you a better understanding of how plugins work. And sometimes, you can get copy the code and modify it for your own use.

• The plugin framework is likely to change in the future. Most of the changes are likely to be additional functions in the API. Function names that begin with "helper_" are candidates for inclusion in the API.
• The NPR doesn't follow the programming paradigm described in Intel books on IXP programming. But quite often, we use the same function names as found in Intel's IXP books.
• The microengine C compiler can sometimes get extremely confused on seemingly trivial code changes.
• The source code will have unfamiliar keywords such as "shared", "__dcl_spec(gp_reg)", "__dcl_spec(local_mem)", and "ctx()". These all have to do with the memory hierarchy and hardware threading that are important aspects of IXP programming. Hopefully, the next few pages will illuminate the semantics and the proper use of such keywords.

The source code of most predefined plugins is organized into the following segments:

• #include statements

Typically, these lines come from copying plugin code that you plan to use as a base. They include the functions from the plugin API. Sometimes developers augment the functions with their own functions. In the examples, these functions are usually collected together in a file called plugin_helpers.h.

• Standard global variable and register declarations

These declarations define variables whose values are often used by support functions and are set as a side-effect of calling functions that send and receive messages from various communication rings in the NPR. They should NOT be deleted.

• The handle_pkt_user(), handle_msg() and callback_user() and plugin_init_user() functions

When writing a new simple plugin, handle_pkt_user() and handle_msg() are the two most likely functions will change. Usually, only a few changes are needed in plugin_init_user(). And callback_user doesn't need to be changed at all unless you need to execute code periodically. This is the case when developing the mycount plugin by modifying either the count or debug plugin code.

• Other functions including plugin initialization and main

You should never have to modify this part of the code when writing a simple plugin. However, when we developed the delay plugin by starting with the mycount, we did have to modify parts of this code. But the delay plugin is not a simple plugin.

We installed the mycount plugin at NPR 1 (left NPR) and directed all packets coming into port 1.1 to go to the plugin. The plugin counts the packet by incrementing two counters and then forwards the packet onto the Queue Manager. It also displays a debug message whenever it gets a packet and responds to control messages from the RLI. It counts packets in two ways: 1) by incrementing PluginCounter 0 which can be easily displayed in a chart; and 2) by incrementing a normal variable whose value can be displayed by sending the plugin a control message requesting the value of the counter.

The figure (above) shows the effect of sending 10 ping packets from n1p1 to n2p1. The chart shows PluginCounter 0 increasing from 0 to 10 and was obtained through the NPR 1 => Monitoring => Plugin Counter menu item. The Plugin Command Log window shows the value of the variable pkt_count which was displayed when we sent a get command to the plugin. The xterm window shows the debug messages that were sent from the plugin a meta-packet was handled by the handle_pkt() routine. These messages were stored in the file debug.log after configuring the debugging by selecting NPR 1 => Plugin Debugging. Note that the packet count is displayed as a hexadecimal number (e.g., 'pkt xa' is referring to packet 10 decimal).

At the beginning of the mycount plugin code (past the initial comment lines), we see the following two lines:

#include "plugin_api.h"
#include "./plugin_helpers.h"

The file ./plugin_helpers.h contain three helper functions:

• helper_plugin_cntr_zero(): Zero a PluginCounter.
• helper_ultohex_lmem(): Convert from unsigned long to string of hexadecimal characters using local memory variables.
• helper_ultodec_lmem(): Convert from unsigned long to string of decimal characters using local memory variables.

The next group of lines we see are the thread-specific global variable declarations and should not be modified:

__declspec(gp_reg) int dlNextBlock;  // where to send packets to next
__declspec(gp_reg) int dlFromBlock;  // where to get packets from
__declspec(gp_reg) int msgNextBlock; // where to send control messages to next
__declspec(gp_reg) int msgFromBlock; // where to get control messages from

// see ring_formats.h for struct definitions
volatile __declspec(gp_reg) plc_plugin_data ring_in;  // ring data from PLC
volatile __declspec(gp_reg) plugin_out_data ring_out; // ring data to next block

__declspec(local_mem) unsigned int timeout; // time (cycles) between callbacks
__declspec(gp_reg) unsigned int pluginId;   // plugin id (0...7)

• __declspec(gp_reg): This is used to declare the storage attribute of the variable. In this case, gp_reg means that a general-purpose register should be allocated for the variable if possible. A few other common storage types are local_mem (local memory), sram (SRAM) and dram (DRAM). But there are others. Later we will see that when the compiler runs out of general-purpose registers, you can tell the compiler how to spill; i.e., what type of memory to use as an alternative. For example, a typical spill rule is to spill from general-purpose registers to local memory and finally to SRAM.
• volatile: This is used to declare that the value of the variable may be changed at any time by code that is unknown to the compiler. This is a property of parallel systems and can happen in the NPR in many ways. But the variable ring_in above defines a structure where the PLC block will place meta-packets which the plugin should read.

Note that the above variables appear to be global (outside the scope of any of the functions) and they are. But also, they will actually be replicated for each thread when the plugin is loaded ... unless they have the shared attribute (which they don't here). The variables are used in the ways summarized in the following table:

For example, dlNextBlock is usually set to the value of QM. QM is a constant that is defined in ~onl/npr/pluginFramework/plugin_dl.h to be 0 and stands for the Queue Manager block.

Note that every thread is loaded with and executes a main() function. The main function does initialization and then executes a code segment based on the hardware context returned by the ctx() call.

	void main()
	{
1	    int c;
	
2	    plugin_init();
3	    dl_sink_init();
4	    dl_source_init();
	
5	    c = ctx();	// get the thread's hardware context# (0-7)
	
6	    if(c >= FIRST_PACKET_THREAD && c <= LAST_PACKET_THREAD)
7	    {
8	    	while(1)	{ handle_pkt(); }
9	    }
10	#ifdef MESSAGE_THREAD
11	    else if(c == MESSAGE_THREAD)
12	    {
13	    	while(1)	{ handle_msg(); }
14	    }
15	#endif
16	#ifdef CALLBACK_THREAD
17	    else if(c == CALLBACK_THREAD)
18	    {
19		while(1)	{ callback(); }
20	    }
21	#endif
	}

• Lines 10, 15, 16, 21: The plugin can be configured (through the Makefile) to have or not have a control message thread or a periodic computation (callback) thread.
• Line 2: Plugin initialization is discussed in the next section. plugin_init() does framework initialization and calls plugin_init_user() to do do user-specific initialization.
• Lines 3-4: dl_sink() is called to forward a meta-packet to the input ring of the next block. dl_source() is called to read a meta-packet from the output ring.
• Line 5: ctx() returns the hardware context, a number between 0 and 7 inclusive.
• Lines 6-21: Depending on the hardware thread context, one of the functions handle_pkt(), handle_msg() or callback() is called. These functions execute an infinite loop and will never return.

The function plugin_init() contains the plugin initialization code that is common to all plugins. It initializes global variables (e.g., pluginId) and then calls plugin_init_user() to allow users to do their own initialization processing.

	void plugin_init()
	{
1	  timeout = 10000;
2	  dlNextBlock = QM;
3	  switch(__ME())
4	  {
5	    case 0x7:
6	      pluginId = 0;
7	      dlFromBlock  = PACKET_IN_RING_0;
8	      msgFromBlock = MESSAGE_IN_RING_0;
9	      msgNextBlock = MESSAGE_OUT_RING_0;
10	      break;
11	    case 0x10:
12	      pluginId = 1;
13	      dlFromBlock  = PACKET_IN_RING_1;
14	      msgFromBlock = MESSAGE_IN_RING_1;
15	      msgNextBlock = MESSAGE_OUT_RING_1;
16	      break;
17	   ... Cases 0x11, 0x12, 0x13, default ...
18	  }
19	  plugin_init_user();	// user hook
	}

• Line 1: The variable timeout will be discussed in Tour Of The Delay Plugin.
• Line 2: Says that when you call dl_sink() to send a meta-packet, put it into the input ring for the Queue Manager.
• Line 3: __ME() returns the actual microengine number that the code is running on. The cases set global variables for that microengine.
• Lines 5-10: Microengine 7 corresponds to plugin ME 0. Plugin ME 0 is expected to read its meta-packets from a particular input ring (dlFromBlock); read its control packets from a particular input message ring (msgFromBlock); and write its replies to control packets to a particular output message ring (msgNextBlock).
• Lines 11-17: These lines serve the same function as lines 5-10 but for the other four plugin MEs.
• Line 19: Allows the user to execute plugin initialization specific to their plugin.

The function plugin_init_user() contains the plugin initialization code that is specific to the plugin.

	void plugin_init_user()
	{
1	    if(ctx() == 0)
2	    {
3		pkt_count = 0;
4		debug_on = 1;
5		sleep(timeout);
6		helper_plugin_cntr_zero( 0 );
7	    }
	}

• Lines 1-8: Only thread 0 should execute the user intialization.
• Line 3: Zero one of the packet counters.
• Line 4: Turn on debugging by default. The d command toggles this variable between 0 and 1.
• Lines 5-6: Zero PluginCounter 0. The sleep() call allows the plugin to complete its initialization before we attempt to zero PluginCounter 0.

The function handle_pkt() is called from main() if the thread context is one of the packet handling threads.

	void handle_pkt()
	{
1	    dl_source_packet(dlFromBlock);
2	    default_format_out_data(dlNextBlock);
3	    handle_pkt_user();
4	    dl_sink_packet(dlNextBlock);
	}

• Line 1: Read a meta-packet from PLC (Parse, Lookup and Classify).
• Line 2: Format the output ring data based on the input ring data.
• Line 3: Call the user's handle packet routine.
• Line 4: Write the meta-packet to the Queue Manager.

The function handle_pkt_user() does the actual packet processing.

	void handle_pkt_user() {
1	    __declspec(local_mem) char dbgmsg[28]	= "got pkt x";
2	    __declspec(local_mem) char pkt_count_str[9];
3	    ++pkt_count;
4	    if( debug_on ) {
5		onl_api_int2str( pkt_count, pkt_count_str );
6		strcat_lmem(dbgmsg, pkt_count_str);
7		onl_api_debug_message(msgNextBlock, dbgmsg);
8	    }
9	    onl_api_plugin_cntr_inc(pluginId, 0);
	}

• Lines 1-2: The variable dbgmsg[28] will contain the debug message and initially contains the C-string "got pkt x". Line 6 (strcat_lmem) will append the formatted value of pkt_count to this intial string. All debug messages are composed of a sequence of C-strings that are limited to 28 bytes including any null characters. Numbers must be formatted into their ASCII representations.
• Line 3: Increment one of the packet counters.
• Lines 4-8: Output a debug message if debug_on is true. onl_api_int2str() converts the value of pkt_count from its internal binary representation to a hexadecimal ASCII string. Let me repeat that in a different way: pkt_count_str will contain the hexadecimal ASCII representation. For example, if the value of pkt_count is 20 decimal, pkt_count_str will contain "14". Line 6 will append "14" to "got pkt x" to produce the debug message "got pkt x14". Line 7 forms the debug message and puts it into the input ring for the Xscale.
• Line 9: Increment the counter in PluginCounter 0.

The function handle_msg() is called from main() if the thread context is the message handling thread. Some plugins call handle_msg_user() to do the plugin-specific processing of control messages, but mycount does not.

	void handle_msg()
	{
1	    __declspec(gp_reg)	  unsigned int message[8];
2	    __declspec(local_mem) char in_msgstr[28];
3-7	    ... other declarations ...
8
9	    for( i=0; i<8; ++i ) { message[i] = 0; }
10	    dl_source_message( msgFromBlock, message );
11
12-14	    ... make a copy of the request message header ...
15
16	    onl_api_intarr2str( &message[1], in_msgstr );
17-31	    ... process the request which should be "z", "d" or "g" ...
32
33-37	    ... form the control message response header ...
38	    dl_sink_message(msgNextBlock, message);
	}

We see the general structure of handle_msg(). Line 10 inputs the message in raw form, and line 38 outputs the response message.

• Lines 1-2: The request message comes to the plugin in raw form and stored as 28 integers (4-bytes each) in message[8]. It is translated into usable C-string form and stored in in_msgstr[28]. This translation is required because of an artifact of the IXP ME architecture.
• Lines 9-16: message[0] contains the message header, and message[1..7] contains the message body. Line 16 does the translation of the message body so that the user can treat in_msgstr[] as a familiar sequence of C-strings.

1	    __declspec(gp_reg)	  unsigned int message[8];
2	    __declspec(local_mem) char in_msgstr[28];
3	    __declspec(local_mem) char pkt_count_str[28];
4	    __declspec(gp_reg)	  onl_api_ctrl_msg_hdr hdr;
5	    __declspec(gp_reg)	  int i;
6	    __declspec(local_mem) char out_msgstr[28];
7	    __declspec(local_mem) char BAD_op_err[8]	= "BAD OP";

However, note that the __declspec() portions of the declarations indicate the variables are to be either in general-purpose registers or local memory. Also, note that the type onl_api_ctrl_msg_hdr is defined as a union between an int and several bit fields. The definition of onl_api_ctrl_msg_hdr is in ~onl/npr/pluginFramework/plugin_dl.h.

Now we discuss the processing that precedes the command processing code. In most cases, these lines should never be changed.

9	    for( i=0; i<8; ++i ) { message[i] = 0; }
10	    dl_source_message( msgFromBlock, message );
11
12	    hdr.value = message[0];
13	    if( hdr.type != CM_CONTROLMSG )	{ return; }
14	    if( hdr.response_requested != 1 )	{ return; }

• Lines 9-10: Line 9 effectively sets all of the bytes in the user's input buffer to the NUL character ('\0') so that message[8] will have no junk bytes after dl_source_message() reads the message from the control message input ring.
• Lines 12-14: The message header is in message[0]. Line 12 makes a local copy of the header and tests a few fields in the header for validity. hdr is actually defined as a union of two structs where hdr.value views the header as an unsigned int and hdr.type and hdr.response_requested make up a struct that overlays hdr.value.

Now we discuss the command processing part. If we look only at the control structure of the code, it is obvious that the code is looking to see if the first character in the message payload is one of the three characters 'z', 'd' or 'g' These characters represent the commands zero counters, toggle debug flag, and get counts. There is currently no standard for the format of control messages other than that they are a sequence of strings. The mycount plugin uses the simplest form of command. The delay plugin uses much longer command names (e.g., "=counts", "delay=", "=delay", "reset") and uses SRAM instead of local memory for processing command names. The one difficulty with using local memory is that there is only 4,096 words of local memory.

17	    if( in_msgstr[0] == 'z' ) {
18		pkt_count = 0;
19		helper_plugin_cntr_zero( 0 );
20	    } else if( in_msgstr[0] == 'd' ) {
21		debug_on = (debug_on+1) % 2;
22	    } else if( in_msgstr[0] == 'g' ) {
23		helper_ultodec_lmem( pkt_count, pkt_count_str );
24		strcpy_lmem( out_msgstr, pkt_count_str );
25		if( onl_api_str2intarr(out_msgstr, &message[1]) < 0 )
26		    { return; } 
27	    } else {
28		strcpy_lmem( out_msgstr, BAD_op_err );
29		if( onl_api_str2intarr(out_msgstr, &message[1]) < 0 )
30		    { return; } 
31	    }

• Lines 18-19: These lines respond to the z command by zeroing the packet counter variable pkt_count and the packet count in the PluginCounter 0 (by calling helper_plugin_cntr_zero()). Note that helper_plugin_cntr_zero() is not in the API (although it will likely in the near future). Note that unlike lines 25 and 29, we do not call onl_api_str2intarr() because the reply message will be empty.
• Line 21: Line 21 just toggles the value of debug_on.
• Lines 23-26: These lines form the response message containing the value of pkt_count.
  Line 23 creates the decimal ASCII form of the packet count in pkt_count_str. (Probably defining pkt_count_str[28] as having 28 bytes is overkill since the number is not going to have more than 9 digits plus the NUL byte.)
  The return call in line 26 only occurs if there was an error returned by onl_api_str2intarr().
  Line 25 puts the output message out_msgstr into the rightmost seven words of message[] so that dl_sink_message() can send back the packet count; i.e., we reuse message[8] as the user's output message buffer.
• Lines 28-30: These lines return the string "BAD OP" for all other commands.

The code that sets up the header of the reply message is straightforward, and you should not have to modify these lines.

33	    hdr.type = CM_CONTROLMSGRSP;
34	    hdr.response_requested = 0;
35	    hdr.num_words = 7;
36	    message[0] = hdr.value;

You should now have a better appreciation of how to write a simple plugin in which the packet handling involves getting a meta-packet, doing some simple counting, and then sending the meta-packet onto the Queue Manager.

There are a number of issues that we have glossed over or just ommitted. We explain the meaning of the lines in the file Makefile shown below to clarify one such issue.

	# set spill based on your spilling needs
	#   8 = spill to Local Memory then SRAM
	#   7 = no spill
	#   6 = spill to SRAM only
	#   5 = spill to Local Memory then Next Neighbor regs
	#   4 = spill to Next Neighbor regs then Local Memory
	#   3 = spill to Local Memory only
	#   2 = spill to Next Neighbor regs only
	#   1 = spill to Next Neighbor regs then Local Memory then SRAM
	#   0 = spill to Local Memory then Next Neighbor regs then SRAM
	SPILL=3

All but the last line are comments. The mycount Makefile has chosen SPILL=3 which means that when the compiler runs out of registers when attempting to allocate registers to variables, it will try to use local memory. And if local memory runs out of space, the compiler will output an error message and quit. This choice can be significant if you were trying to write a larger plugin because the local memory in each microengine contains only 640 32-bit words. Luckily, this is not a problem with the mycount plugin.

One reason for choosing the SPILL=3 option is that the plugin will not use additional SRAM. Later, we will see that the delay plugin does use SRAM, and we will have to change the Makefile to avoid SRAM reference conflicts if we plan to load the plugin onto more than one ME. Because the mycount plugin doesn't use additional SRAM, we could load the mycount plugin into any or all of the five plugin MEs without worrying about SRAM usage by the compiler.

We now need to cover three topics to give a more complete view of NPR plugins:

• How to do more complicated (although still basic) processing.

  For example, if we use an auxilliary filter to send a copy of a meta-packet to a plugin for accounting purposes, we will want to drop the packet after we have done updating counters. These processing functions include dropping a packet, reading various protocol headers, reading the packet payload, modifying a packet, or copying a packet.

• How to delay a packet where meta-packets need to be queued until a future time.

For example, if we want to implement a different packet scheduling protocol without changing the Queue Manager, we may need a plugin that stores packets into its own queues until it is time for the packet to be transmitted.

• How to test a plugin.

  We showed how to output debug messages. But this approach is the most basic approach to testing and is limited to low rate traffic (e.g., 1-3 packets per second) since the debug processing represents a significant amount of overhead.

NPR Tutorial >> Writing A Plugin	TOC