ETOMIC advanced network monitoring system for future Internet

1. Introduction

The European Traffic Observatory Measurement Infra-struCture (ETOMIC) was launched in 2004 [1,2]. It is tar-geted to provide the scientific community with an Inter-net measurement platform that is fully open and recon-figurable, extremely accurate and GPS-synchronized.

The ETOMIC system has been designed to allow re-searchers to perform any kind of active network mea-surement. The users are provided with a web-basedgraphical user interface for the definition of the expe-riments to run. Researchers may either choose from anumber of built-in measurement scripts that cover themost popular measurement techniques, like tracerouteor packet-pair experiments, or they can provide theirown code for the experiments. To avoid conflicts in re-source utilization each measurement has to be sche-duled to exclusively reserve node resources for its exe-cution. The node reservations are performed through theweb-based user interface. The ETOMIC management ker-nel takes care of the software upload and experimentexecution in a fully automated fashion.

After the successful duty of the measurement nodessince 2004 the renewal of the system components wasnecessary. In the Onelab project [3] we have extendedthe capabilities of the measurement hardware to matchthe current technological level and to incorporate thesoftware evolution of the last years that are importantfrom the perspective of network measuerments. TheETOMIC infrastructure now provides two ways of col-lecting experimental data. One possibility is when theresearcher reserves and configures the measurementnodes and sets the parameters of the experiment throughthe Central Management System.

In this case, besides the original ETOMIC nodes,newly deployed enhanced measurement boxes canalso be used for experimentation. To meet the require-ments of high precision measurements the nodes areequipped with a DAG card (for the original nodes) or anARGOS card (for the new generation nodes) to providenanosecond-scale timestamping of network packets.Besides these nodes a third type of hardware compo-ment was also introduced, which is called Advance Prob-ing Equipment (APE). APE is a low cost hardware solu-tion developed to serve as a measurement agent foruser applications: it provides a web service interface toconduct experiments.

This approach enables autonomic software compo-ments to automatically collect relevant network datafrom the ETOMIC system they rely on for their opera-tion. As a consequence of a development in the sys-tem kernel the nodes of the PlanetLab platform [4] canalso be used as measurement nodes by the ETOMIC sys-tem. The goal of this integration was to enable the fede-rated usage of the high precision ETOMIC nodes and thenumerous PlanetLab nodes.

To make it easier to handle and archive the hugeamount of data collected by the ETOMIC platform wehave created data repositories. There are two differentinterfaces for these data archives. The periodic mea-surements web interface can be used to poll automati-cally collected measurement data through pre-definedqueries. As another approach, the Network Measure-ment Virtual Observatory (nmVO) [5] provides standardSQL database access to the user community. The nmVOprovides a graphical user interface and a web serviceinterface for the users to access raw and evaluated mea-surement.

VOLUME LXV. • 2010/I 25

Keywords: network monitoring, ETOMIC, traffic measurement, GPS

ETOMIC is a network traffic measurement platform with high precision GPS-synchronized monitoring nodes. The infrastructure is publicly available to the network research community, supporting advanced experimental techniques by providing high precision hardware equipements and a Central Management System. Researchers can deploy their own activemeasurement codes to perform experiments on the public Internet. Recently, the functionalities of the original system were significantly extended and new generation measurement nodes were deployed. The system now also includes well structureddata repositories to archive and share raw and evaluated data. These features make ETOMIC as one of the experimental facitlities that support the design, development and validation of novel experimental techniques for the future Internet. In this paper we focus on the improved capabilities of the management system, the recent extensions of the node architectureand the accompanying database solutions.

� INVITED PAPER

ETOMIC advanced network monitoring system for future Internet experimentation

ISTVÁN CSABAI, ATTILA FEKETE, PÉTER HÁGA, BÉLA HULLÁR, GÁBOR KURUCZ, SÁNDOR LAKI, PÉTER MÁTRAY, JÓZSEF STÉGER, GÁBOR VATTAY

Eötvös Loránd Univeristy, Budapest, Hungary {csabai,fekete,haga,hullar,kurucz,laki,matray,steger,vattay}@etomic.org

2. System architecture

The ETOMIC infrastructure is constituted of high preci-sion measurement equipment modules placed hostedby European universities, research institutes and com-pany laboratories. The clocks of the measurement nodesare synchronized via GPS signals, which allow not onlypacket round-trip time estimation, but also precise one-way delay measurements. The ETOMIC platform is veryflexible, since researchers can develop and run anykind of active experiments.

A Central Management System (CMS) is in charge ofsystem control, comprising not only the scheduling andexecution of measurements experiments, but also sys-tem monitoring and configuration. The main softwarecomponent of the CMS is called the management kernel

which is running on a dedicated server computer. Thekernel is responsible for scheduling tasks, deploymentof user software to the measurement nodes, node con-figuration, experiment execution and the collection ofmeasurement results from the nodes. The CMS providesa web-based graphical user interface where the research-ers can configure the system and reserve system re-sources for their measurements. An internal databaseis attached to the kernel where the system and userlevel management information are stored. The resultsof a finished measurement are also collected and storedin this database until the user downloads them. The sys-tem components are depicted in Fig. 1.

As an important add-on for the original ETOMIC sys-tem the architecture has been extended with large ca-pacity data repositories that are publicly available. The

INFOCOMMUNICATIONS JOURNAL

26 VOLUME LXV. • 2010/I

Figure 1. ETOMIC system architecture

Figure 2. Scheduling of experiments and maintenance tasks

system provides several different interfaces for theserepositories through which the users can reach the col-lected datasets. The interfaces allow the users to runintelligent queries in order to filter and process the rawdata on the server side.

3. Management kernel

3.1 Central Management System (CMS)The ETOMIC management kernel constitutes the core

of the Central Management System (CMS). It is in chargeof user management, experiment scheduling and keep-ing the corresponding results in the temporary datastorage.

In order to isolate the different measurements and toschedule experiments and maintenance tasks a calen-dar is used for each measurement node, as it is shownin Fig. 2. The management tasks can be divided intotwo branches: tasks that correspond to experiment de-finitions and tasks that correspond to the execution ofthe scheduled experiments.

3.2 Scheduling and calendar maintenanceThe researcher is expected to book measurement

nodes for a certain time interval and to upload the app-lications neccessary for the measurement. The web in-terface is in charge of checking that the timeline for theexperiment does not collide with any other previously

registered measurement. In case of a successful re-source reservation the CMS inserts the new experimentinformation into its internal database. The managementkernel is continuously checking the internal databasefor new measurement requests. Once a new experimenthas been defined and the deadline for execution app-roaches the management kernel uploads the measure-ment softwares and configures the nodes, executes theexperiment and finally downloads the results to its tem-porary storage.

3.3 Web interface to CMSUsers are provided with a graphical interface for set-

ting up the experiment beforehand. Then, the manage-ment kernel is in charge of experiment execution. Aninternal database is used to store all the necessary in-formation to run the experiments. The stored informa-tion includes the applied softwares, external data files,the experiment results, the experiment status and themeasurement node status.

Users can find manuals, the programming API andexample codes here. The users can do the following ope-rations: add a new program; upload the necessary datafiles; define the experiment bundle by scheduling thestart and end times of the measurement; book ETOMICtime and reserve the measurement nodes; downloadthe results; define periodic experiments with the repe-tition period; sharing files with other users.

ETOMIC advanced network monitoring system


Figure 3. List of experiment bundles on the web based graphical user interface

3.4 Integration of Planetlab’s nodesPlanetLab is a global platform for supporting the

development of new network services. This platform isalso used for network experiments. The nodes of the Pla-netLab platform are accessed interactively via remoteshell. This access method enables the CMS to use thePlanetLab nodes as its own nodes. Although the mainhardware capabilit ies of the PlanetLab and ETOMICnodes significantly differ, the large number of PlanetLab nodes makes them very attractive to the user com-munity.

The capabilities of PlanetLab are not described inthis paper, here we only note that the PlanetLab nodesare usually up-to-date server PCs without any hardwarecomponents specialized for network measurements. Theslice based management of PlanetLab nodes allowsmultiple users to run experiments simultaneously in thesame remote node at the same time, while the CMS takescare of the unique resource allocation. In spite of thebasic differences of PlanetLab and ETOMIC the federa-ted usage of the high precision ETOMIC nodes and thenumerous PlanetLab nodes could lead to new ways ofexperimentation.

The software installed on ETOMIC nodes has beenadapted to make the joint usage possible, using a sliceof PlanetLab that is automatically renewed by the CMS.This makes the whole range of ETOMIC and PlanetLabremote nodes available through the ETOMIC web inter-face. The most important challenge for the integration

was the synchronization of the clocks in nodes fromboth platforms as they use different reference signalswith highly different precisions.

4. The measurement nodes

The nodes can be divided into two groups based on theirhardware architecture. The ones that are built on serverPC architecture are called ETOM. These nodes are ac-cessible via the web-based graphical interface present-ed in Section 3.3. The ones that are based on a light-weight programmable board are called APE. The APEnodes are accessible via a web service interface. GPSreceivers are connected to all types of measurementhardware to provide the precise time synchronizationbetween the nodes and to provide the reference clock forthe measurement cards.

Both of ETOM’s and APE’s hardware solutions is high-precision ones, due to the incorporated precision equip-ment (DAG and Argos cards) that are specifically de-signed to transmit packet trains with strict timing, in therange of nanoseconds.

APE is built on a development board with Blackfinprocessor. The board is manufactured by Analog Devi-ces Inc. and has a number of different interfaces for host-ing auxiliary hardware components that are responsiblefor specific network measurements tasks. Each mea-surement node is provided by two network interfaces:



Figure 4. Hardware components of ETOMIC responsible for the precision timestamping of the network packets. From left to right: the Endace DAG 3.6GE measurement card,

the netFPGA based ARGOS measurement card and the APE node.

Table 1. Available measurement nodes in the ETOMIC system *under deployment, **under integration

a standard network interface card for management pur-poses (maintenance, software upload, data download)and an additional precision card for the probe traffic.The main feature of the APE nodes is that they providemeasurement services which can be remotely calledby user applications in an online fashion, without timeslot reservation.

5. Accessing measurement data

5.1 Data repositoriesFor network measurements the collected raw data

is traditionally stored in files in some standard (like tra-ceroute dump, tcpdump) or custom formats. The filesare then processed according to the research ques-tions to be answered. Detailed analysis of complex net-works requires large statistical samples. This require-ment leads to substantial data size in case of mea-surements in high bandwidth networks, even if just afew parameters of the packets are recorded (like IP add-resses, arrival time, protocol, size or delay).

Practically, measurements can produce dozens ofmegabytes at each monitoring node that sums up tohundreds of megabytes or even terabytes in multi-nodeexperiment. Keeping only the results of the data analy-sis and discarding the raw data themselfes is not agood way to solve the data handling issues: measure-ment data gathered today cannot be reproduced in thefuture. Thus it is preferable to store the original data-

sets to allow further re-analysis and support the studyof the long-term evolution of the network.

For these purposes we have created data reposito-ries to store measurement data collected by means ofthe ETOMIC system. There are two different interfacesto reach raw and aggregated measurement data. Theperiodic measurements web interface can be used topoll data collected from automatic measurements in thesystem. The users can choose from pre-defined que-ries by selecting a given type of measurement, a set ofnodes on which the measurement was conducted anda time frame. Long term data sets are available for one-way delay values, traceroute measurements and Paristraceroute measurements [6] data between all the ETOM-IC node pairs. The results of these measurements arereachable through the periodic measurement reposito-ry of the ETOMIC website’s Database / Open Repositorymenu [1].

5.2 Network Measurement Virtual Observatory (nmVO)The basic concept presented in [5] is an approach to

efficiently store and share research data. Beyond thesimple data collecting and archiving functions it aimsat providing easy-to-use analysis tools via both humanand machine readable interfaces.

One of the main features of the nmVO is that it pro-vides SQL access to the databases that are integratedunder its framework, thus the users can edit and run theircustomized queries through either the web-based SQLinterface or the web services interface. The main ad-



Figure 5. The web interface of the nmVO

vantage of this solution is that the researchers can fil-ter out the relevant information from the huge archivesusing server side processing. Hence, only the neces-sary datasets and results have to be downloaded fromthe server.

To sketch the nmVO principle through a possible app-lication, consider a scenario in peer-to-peer overlay net-works where management information is needed to op-timize the routing between the peers. It would be un-thinkable to use gzipped files for such real-time evalua-tion. On the contrary, the scenario is feasible if one turnsto the nmVO to get the typical loss rate, the average de-lay on certain routes or the shortest path between thepeers. This means that beyond the data itself, analysistools are also needed to perform such data filtering andtransformation queries efficiently. Using these storedprocedures we can move the typical filtering and pre-processing tasks to server side.

The majority of the experimental data collected inthe ETOMIC system is inserted into the data reposito-ries and can be reached through both the nmVO webgraphical user interface and the web service interface.The developed nmVO API is integrated into the APEnodes’ software, so that all measurement data from theAPE boxes are automatically copied into the nmVO datarepository.

In addition to the historical raw data collections andthe evaluated results of periodic measurements, also non-ETOMIC traceroute logs and topology data [7], one-waydelay values [8], queueing delay tomography data [9],available bandwidth results [10], router interface cluster-ing and IP geolocalization data [11] can be found in thearchive.

The nmVO can be accessed through the ETOMIC web-site’s Database / CasJobs Query Interface menu [1] andalso via Web services for client applications.

6. Conclusions

In this paper we presented the enhanced ETOMIC net-work measurement infrastructure. We described thekey components of the architecture and the new fea-tures of the Central Management System. The improvedsystem kernel includes support for periodic measure-ments and the federated usage of the high precisionETOMIC nodes and the numerous PlanetLab nodes. Be-sides the kernel development novel hardware compo-nents have been developed and deployed. New light-weight measurement equipments have been installedthat provide measurement services which can be re-motely called by user applications via web services.

The system now also includes well structured datarepositories to archive and share the experimental data.Periodic measurement data can be polled with customiz-able pre-defined queries, while the nmVO frameworkgives full SQL access to its archive. The recent develop-ments make ETOMIC an easy to use experimental facit-lity with versatile feature for network research.

Acknowledgements

The authors thank the partial support of the EU ICTOneLab2 Integrated Project (grant agreement No.224263),

the EU ICT MOMENT Collaborative Project (grant agreement No.215225) and the National Office

for Research and Technology (NAP 2005/ KCKHA005).

Authors

ISTVÁN CSABAI is a professor of physics at the De-partment of Complex Systems, Eötvös Loránd Univer-sity. His research interest is quite wide, ranging fromartificial intelligence through cancer research to cos-mology. Although these disciplines are diverse, theway of handling of their inherent complexity and thenecessary computational tools are often common.He has written his first paper about the dynamics ofcommunication networks 15 years ago, recently heis working on the development of the ETOMIC net-work measurement system and the Network Measure-ment Virtual Observatory concept.

ATTILA FEKETE was born in 1975, Budapest. He re-ceived his MSc degree in physics from Eötvös Lo-ránd University (ELTE) in 1999. After graduation hestudied TCP dynamics in collaboration with ELTE Com-munication Networks Laboratory and Ericsson Re-search. Between 2003 and 2008 he worked in Colle-gium Budapest Institute for Advanced Studies on themodeling of complex networks. He received his PhDdegree in physics from ELTE, where he is currentlyworking as a teaching assistant, in 2009.

PÉTER HÁGA is an assistant professor at the Eöt-vös Loránd University. He received his PhD degreein physics in 2008. From 1999 he is a permanentmember of the Communication Networks Laboratoryand he is one of the founders of the ETOMIC mea-surement infrastructure. His research interest in-cludes network measurement techniques, traffic mo-deling and adaptive protocols.

BÉLA HULLÁR received his MSc diploma in Compu-ter Science in 2007 from the Eötvös Loránd Univer-si ty. He has been working in ELTE CommunicationsNetwork Laboratory since then, as a PhD student.His research interests are in monitoring and mea-suring computer networks, developing the appropri-ate measurement infrastructure and its managementsystem.

GÁBOR KURUCZ received his first MSc degree inTelecommunications and Media Informatics at 2003from the Budapest University of Technology and Eco-nomics (BME). He earned his second diploma in Se-curity of Information and Communication Systems at2008 from BME. In 2006, he joined the Department ofPhysics of Complex Systems at ELTE. His researchinterests are in developing anetwork equipment witha precise timestamping module under the OneLabproject.

SÁNDOR LAKI received his MSc degree in computerscience from the Eötvös Loránd University, Budapest,in 2007. Currently he is working towards his PhD incomputer science at the Department of Physics ofComplex Systems. His primary research interest in-cludes Internet measurement techniques, adaptiveprotocols and network modeling, especially IP geo-location and traffic classification.



PÉTER MÁTRAY received his M.Sc. degree in mathe-matics and computer science from Eötvös Loránd Uni-versity, Budapest in 2005. Currently he is doing hisPhD studies at the Department of Physics of Comp-lex Systems. His attention is mainly focused on thedatabase aspects of network measurements. Besidesthat, he is partly involved in active probing Internetmeasurements (especially network tomography) andvisualization.

JÓZSEF STÉGER graduated as a biophysicist at theFaculty of Science at Eötvös Loránd University in 2001.Now works as an assistant lecturer at the Dept. of Phy-sics of Complex Systems. He is carrying out researchtasks within the study of communication networks,like the Internet. By using the measurement archi-tecture introduced in the article he made queuingdelay tomography experiments possible, which en-ables the group to notice congested hotspots withinthe network. He is an active member of internationalprojects Moment and Onelab2.

GÁBOR VATTAY is a professor of physics of comp-lex systems at the Faculty of Sciences of Eötvös Uni-versity. He received his PhD in 1994 and his DSc titlein 2003 in mathematical physics of chaotic systems.His interest is in the complex dynamics arising in na-tural and man-made systems. Since 1998 his inter-est shifted toward the dynamics of networked sys-tems. He is the founder and director of the Commu-nication Networks Laboratory established at EötvösLoránd University by Ericsson Research since 2000and the leader of the EU supported Internet measure-ment effort ETOMIC.

References

[1] The ETOMIC website, http://www.etomic.org

[2] D. Morato, E. Magana, M. Izal, J. Aracil, F. Naranjo,F. Astiz, U. Alonso, I. Csabai, P. Haga, G. Simon, J. Steger, G. Vattay, ETOMIC: A testbed for universal active and passivemeasurements. IEEE TRIDENTCOM 2005, Best Testbed Award, pp.283–289, 23-25 Febr. 2005.

[3] OneLab – Future Internet Test Beds,http://www.onelab.eu

[4] L. Peterson, T. Anderson, D. Culler, T. Roscoe, A Blueprint for Introducing Disruptive Technologyinto the Internet. Workshop on Hot Topics in Networks,October 2002.

[5] P. Mátray, I. Csabai, P. Hága, J. Stéger, L. Dobos, G. Vattay, Building a Prototype for Network Measurement VirtualObservatory. ACM SIGMETRICS – MineNet 2007, 12 June 2007, San Diego, USA.

[6] B. Augustin, X. Cuvellier, B. Orgogozo, F. Viger, T. Friedman, M. Latapy, C. Magnien, R. Teixeira,Avoiding traceroute anomalies with Paris traceroute.Internet Measurement Conference, October 2006.

[7] S. Garcia-Jimenez, E. Magana, D. Morato, M. Izal,Techniques for better alias resolution in Internet topology discovery. IFIP / IEEE IM 2009, 1-5 June 2009.

[8] A. Hernandez, E. Magana, One-way Delay Measurement and Characterization.International Conference on Networking and Services (ICNS ‘07) , 2007.

[9] T. Rizzo, J. Stéger, I. Csabai, G. Vattay, P. Pollner,High Quality Queueing Information from AcceleratedActive Network Tomography. Tridentcom 2008,Innsbruck, Austria, March 17-20, 2008.

[10] P. Hága, K. Diriczi, G. Vattay, I. Csabai, Granular model of packet pair separation in Poissonian traffic. Computer Networks, Vol. 51, Issue 3 , pp.683–698,21 Februar 2007.

[11] S. Laki, P. Mátray, P. Hága, I. Csabai, G. Vattay, A Detailed Path-latency Model for Router Geolocation.IEEE TridentCom 2009 Conference, 6-8 April 2009, Washington D.C., USA, 2009.



ETOMIC advanced network monitoring system for future Internet

Documents

Transcript of ETOMIC advanced network monitoring system for future Internet