The  Grid  Observatory   Initiative   develops   a   scientific   view  of   the   dynamics   and  usage  of  globalized  IT  systems  by  monitoring  and  analyzing  the  EGI  grid.    

The   overall   goal   is   to   create   a   full-­‐fledged   digital   curation   process,   with   its   four  components:  preservation,  validation,  indexation  and  knowledge  building.    

As  the  largest  non-­‐profit  globalized  system  worldwide  and  with  demanding  scientific  users,   the   EGI   infrastructure   is   one   of   the   most   exciting   artificial   complex   systems.  

With   extensive   monitoring   facilities   already   in   place,   it   offers   an   unprecedented   opportunity   to   observe   and   to  understand  the  computing  practices  within  the  e-­‐Science  community.    

Grid   and   cloud   share   a   common   paradigm:   they   are   globalized   at   a   large   scale.   As   such,   the   data   collected   and   the  knowledge  built  from  analyzing  EGI  concern  cloud  modeling  as  well.  Ongoing  work  integrates  monitoring  data  from  the  StratusLab  cloud.  

The  Grid  Observatory  is  an  open  collaboration,  keen  to  foster  dialog  and  partnerships  with  others  in  the  relevant  areas  of  computer  science  and  engineering.  The  Laboratoire  de  Recherche  en  Informatique  and  Laboratoire  de  l’Accélérateur  Linéaire,   from  CNRS   and  University   Paris-­‐Sud,   along  with   the   London   Imperial   College   operate   data   production.   The  initiative  is  supported  by  France-­‐Grilles,  INRIA  and  CNRS.    

A  trove  of  experimental  data:  www.grid-­‐observatory.org  

 

The   first   role   of   the   Grid   Observatory   is   to   preserve   the  monitoring  data,  normally  discarded  after  operational  usage,  and   to   make   them   available   to   the   wider   scientific  community.   Through   its   web   portal,   the   Grid   Observatory  offers  public  access  to  a  repository  of  grid  traces  to  observe  e-­‐Science  practice  and  infrastructure.  

• EGI   provides   an   accessible   approximation   of   the  current  and  future  requirements  of  e-­‐Science  users.    

• Grid   status   and   middleware   activity   are   recorded.  These   can   be   explored   for   a   wide   range   of  motivations,   from  operational  usage,  e.g.   improving  performance,   to   scientific   research,   e.g.   testing  classification  methods  for  fault  detection.  

The   Grid   Observatory   follows   Tim   Berners   Lee’s   recommendation   for   Raw   Data   Now.   It   exemplifies   the   Big   Data  challenges:   semantic  organization,  provenance,   interoperability,   and  next  generation  analytics.  Emerging   technologies  such  as  Linked  Open  Data  will  be  explored  to  further  address  those  challenges.    

The  Green  Computing  Observatory      

 The   Grid   Observatory   offers   extensive   traces   of   energy  consumption.   Because   green   IT   is   becoming   an   increasingly   urgent  need  and  also  because  there  was  no  existing  EGI  monitoring  tool,  this  action  has  its  own  name:  the  Green  Computing  Observatory.    

• The   traces   integrate   motherboard-­‐level   monitoring   with  information  on  computing,  networking,  storage,  and  cooling.  

• Acquisition  exploits  the  de  facto  standards  IPMI  and  Ganglia.  

• Integration   is   based   on   an   ontology   of   IT   system  measurements,   including   virtual   machines,   developed   by  University  Picardie  Jules  Verne.    

 

From  applied  to  fundamental  research  

Research  exploiting  the  monitoring  data  should  demonstrate  verifiable  and  positive  impact  on  production  systems.    • Beyond-­‐power-­‐law   and   non-­‐stationary   behavior   are   pervasive.   With   sequential   testing,   segmentation   and  

adaptive  on-­‐line  clustering,  we  advanced  fault  detection  and  parsimonious  model  building.  • Efficient   autonomic   policies   must   combine   a   priori   knowledge   and   on-­‐line   adaptation,   but   reference  

interpretations   are   most   often   missing.   Data-­‐driven   topic   modeling   in   the   spirit   of   text   mining,   and  heterogeneous  data  integration  with  Statistical  Relational  Learning  help  to  build  intelligible  representations.    

         

Digital  curation  

The  overall  goal  of  the  Grid  Observatory  is  to  create  a  full-­‐fledged  digital  curation  process,  with  its  four  components.  

Establishing  and  developing  a  long-­‐term  repository  of  digital  assets  for  current  and  future  references.  

The  Grid  Observatory  operates  since  October  2008.   It  continuously  records  and  publishes  various  traces.  An  essential  achievement  is  to  cover  the  complete  scope  of  the  grid  middleware  and  users  activity,  beyond  particular  aspects  such  as  job  lifecycle  or  failure  events,  and  including  for  instance  logging  the  Information  System  (BDII).  

Providing  digital  asset  search  and  retrieval  facilities  to  scientific  communities  through  a  gateway.  

The  middleware  traces  are  currently  made  available  only  in  raw  format,  on  a  weekly  basis.  Much  remains  to  be  done  in  the   direction   of   a  more   semantic   organization.   The   Green   Computing   Observatory   data   are   organized   along   an   XML  schema  associated  with  the  measurement  ontology.  All  are  available  trough  the  Grid  Observatory  portal.    

Tackling  the  good  data  creation  and  management  issues,  and  interoperability,  through  formal  ontology  building.  

The   Grid   Observatory   most   often   builds   on   EGI   and   gLite   monitoring,   thus   benefits   from   their   collective   effort   of  middleware   development   and   EMI   standardization.   The   Green   Computing   Observatory   builds   on   IPMI   and   Ganglia.  Calibration   of   IPMI   measurements   is   made   possible   by   PDU   (Power   Distribution   Unit)   measurements.   The   Green  Computing  Observatory  participates  in  the  COST  action  IC0804  -­‐  Energy  efficiency  in  large  scale  distributed  systems.    

Adding  value  to  data  by  generating  new  sources  of  information  and  knowledge  through  semantic,  statistical  and  Machine  Learning  based  inference.  

The   general   framework   for   the   Grid   Observatory   is   to   turn   it   into   a   social   intelligence   system   to   pool   scientific   and  engineering   expertise,   in   order   to   build   gradually  more   integrated  models   of   the   European   e-­‐infrastructures,   and   to  define  and  validate  autonomic-­‐oriented  policies  addressing  their  operational  challenges.  

More  information:    

• The  Green  Computing  Observatory:  a  data  curation  approach  for  green  IT.  9th  IEEE  Int.  Conf.  on  Dependable,  Autonomic  and  Secure  Computing.    

• The  Grid  Observatory.  11th  IEEE/ACM  Int.  Symp.  on  Cluster,  Cloud  and  Grid  Computing.  

Towards  Open  Linked  Data  

***  Data   are   accessible   on   the   web  through   the   portal;   the   only  protection   implemented   is   against  malicious  usage.  

All   formats   are   machine   readable  and  open:  ASCII,  XML,  SQL,  LDIF      RDF   and   Linked   RDF   are   the   next  step.  

   

Selected  contributions  from  the  Grid  Observatory  initiative  and  its  users  

Fault  detection  and  diagnosis,  smart  probing.  

Distributed  Monitoring  with  Collaborative  Prediction.  12th  IEEE/ACM  Int.  Symp.  on  Cluster,  Cloud  and  Grid  Computing.  

Toward   Autonomic   Grids:   Analyzing   the   Job   Flow   with   Affinity   Streaming.   15th   ACM   SIGKDD   Conf.   on   Knowledge  Discovery  and  Data  Mining.  

Optimization   of   jobs   submission   on   the   EGEE   production   grid:   modeling   faults   using   workload.   Journal   of   Grid  Computing  ,  8(2).  

Grid  models  

Characterizing  e-­‐science  file  access  behavior  via  latent  Dirichlet  allocation  .  4th  IEEE/ACM  Int.  Conf.  on  Utility  and  Cloud  Computing.    

Towards  non-­‐stationary  Grid  models.  Journal  of  Grid  Computing,  9(4).    

Autonomic  Quality  of  Service  and  Green  Computing  

Multiobjective  reinforcement  learning  for  responsive  grids.  Journal  of  Grid  Computing  8:3..    

Autonomic  policy  adaptation  using  decentralized  online  clustering.  7th  IEEE/ACM  int.  conf.  on  Autonomic  computing.