Comprehensive end-to-end and top-to-bottom monitoring is critical for developing and debugging high performance, distributed applications. However, this service is largely unavailable to the application developer except in testbed environments. Increasingly the approach of these applications is to rely on "automatic" tuning of transport parameters such as TCP window size, parallel streams, etc. However, the results of the tuning still must be verified, and sometimes debugged, both of which rely on fine-grained network monitoring. In addition, end-to-end approaches are limited in their ability to diagnose problems in the intervening networks and to diagnose the impact of tuning on other traffic in the network. The information from the monitors will be directly available to applications to aid in debugging and tuning of application data transmission.

Applications will be able to send "request" packets to automatically activate monitoring along the network path between communicating endpoints. The request packets pass through passive sensors that are deployed at the ingress and egress routers of the wide-area networks and at critical points in the end site networks. To activate monitoring, an endpoint of a data stream runs a program that sends request packets to the other endpoint. The goal of these packets is to alert each monitor in the interior of the network that the corresponding application flow is requesting monitoring from the network. Once activated, the monitors open a connection to a remote agent. The sensors will send to the agent a stream of monitoring data extracted from the packet flow. We will be deploying this system at critical ESnet ingress and egress sites and at a few prototype end sites. This passive monitoring system will provide an essential component in a complete end-to-end network test and monitoring capability and will complement the existing network operation efforts. Most critically, this monitoring system will provide a mechanism for applications to determine what is happening to their data in the network. It is expected to be critical in helping to bridge the gap between network engineers and application designers/users.

The goals of this project are:

• An infrastructure that is available to applications and securely enables routine monitoring through a network domain (ESnet and end sites in this case) in order to address:
• Debugging of distributed application communication
• Evaluation of application transport tuning strategies
• Evaluation of the impact on overall network performance of transport tuning strategies
• Support for both active and passive monitoring
• Support for both unicast and multicast traffic monitoring
• A monitoring infrastructure that
• Is self-configuring with respect to requests to activate monitoring
• Introduces traffic only in the form of monitoring data and requests for monitoring
• Provides for access control on both monitoring requests and the data archives
• Investigation of the Grid Forum monitoring architecture (GMA), both for data generated by the approach described here and data that is collected by other approaches
• An authorization framework that is sufficiently flexible that various approaches can be implemented, used, and evaluated
• A monitoring infrastructure that is general enough to be applicable to any network environment

• Design and implement a secure sensor activation protocol
• Deploy active monitoring tools (nettest, NWS, etc.) at a set of sites, in collaboration with Net100
• Design and begin implementation of passive sensor improvements

• Deploy prototype passive monitoring system in ESnet using tcpdump, NetLogger for archiving and manual configuration mechanisms
• Deploy implementation of the Grid Monitoring Architecture for the collection and publication of monitoring data
• Begin deployment of the secure sensor activation protocol implementation
• Deploy initial implementation of passive sensor improvements
• Expand number of passive monitors and active monitoring sites
• Integration with Net100 monitoring data archive (NAIB, from Rich Wolski)

• Begin design and implementation of Berkeley Packet Filter (BPF) enhancement to improve handling of large numbers of flows

• Begin working with application developers running applications across ESnet
• Extend tcpanaly tool to provide real-time flow monitoring

• Continue to work with Applications, and determine what type of monitoring is most beneficial to applications

• Continue design and implementation of Berkeley Packet Filter (BPF) enhancement to improve handling of large numbers of flows

• Continue passive sensor improvements
• Expand number of passive monitors and active monitoring sites
• Explore scalability issues, and enhance tools where necessary

This project will work closely with the net100 project, and in particular plans to utilize the monitoring data archive that net100 will deliver.