“Internet2  Presenta,on”  Brian  Stengel,  Director  of  Opera-ons,  Kinber  Peter  J.  Heverin,  Project  Manager,  Kinber  

Mike  Carey,  Network  Engineer,  Kinber  Jon  Paul  Herron,  Director  of  Engineering,  GlobalNOC  

Luke  Fowler,  Senior  Manager,  Systems  Engineering,  GlobalNOC  Marianne  Chitwood,  Director  of  Opera-ons,  GlobalNOC  

September  20th,  2012  

What  is  PennREN  

•  Capital Budget - $128,958,031 –  Federal Stimulus Funds - $99,660,678 –  Matching Funds - $29,297,353

•  Outside Plant Infrastructure Constructed for PennREN –  48 Stands of NZD Fiber optic Cable –  1700± Route Miles –  Outsourced Fiber Maintenance

•  13 Optical Regeneration Service Nodes •  56 Service Distribution Access Nodes

a high-speed, state-wide, research & education serving healthcare, ���K-20 and the public good

Project  Route  

Total PennREN Fiber route is estimated at 1,613 miles. The route consists of the following;

•  1,086 miles of new aerial construction. •  486 miles of aerial overlash. •  14 miles of new underground construction. •  27 miles of leased underground conduit.

Network Backbone Engineering is 95% complete overall.

3  

Project  Route  

4  

PennREN  Loca,ons  Service  Nodes  

Lehigh  University  

401  N.  Broad  

Penn  State  Hershey  Medical  Center  

Indiana  University  of  Pennsylvania  

Allegheny  Center  Mall  

Slippery  Rock  University  of  Pennsylvania  

Penn  State  Erie  (Behrend)  

Clarion  University  of  Pennsylvania  

Penn  State  Dubois  

Penn  State  –  State  College  

Windstream  

Bucknell  University  

University  of  Scranton  

Amp  Nodes  East  Stroudsburg  University  of  Pa  

Kutztown  University  of  Pa  

West  Chester  University  of  Pa  

Millersville  University  of  Pa  

Shippensburg  University  of  Pa  

PiQ  Johnstown  

Community  College  of  Beaver  Co  

Allegheny  College  

PiQ  Titusville  

Penn  State  Hazelton  

Access  Nodes  46  Loca,ons  on  PennREN  fiber  

Project  Teams  

•  Hardware  procurement  •  Design  •  Integra,on  •  Project  Management  •  Professional  Services  

•  Plan/Design  •  Op,cal  engineering  •  Field  services  •  Professional  services  

•  Staging  •  Shipping  •  Configura,on  •  Tes,ng  •  Turn-­‐up  •  Field  Services  

•  Plan/Design  •  Consul,ng  •  Engineering  •  Support  •  Training  

•  Fiber  OSP  •  Building  entrances  •  Cabinets,  Installa,on  •  Splicing  •  Tes,ng  •  Last  mile  solu,ons  

•  Earthlink  sourcing  for  OOB  MPLS  Network  

•  Last  mile  solu,ons  

•  Service  Desk,  Tier  1,  Tier  2  •  Systems  and  Network  Tools  •  Network  Engineering  •  Net,  Perf,  Change  Management  •  Opera,ons  support  

Node  Installa,on  Status  

Segment  in  Produc,on  

Equipment  Installed  

Under  Construc,on  

Co-­‐Loca,on  at  Member  Sites  

Construc,on  Diagrams  

Op,cal  Core/DWDM  System  

Packet  Core  –  MPLS  

Plan  –  Build  –  Integrate  

Peering  Points  (Ini,al)  

Internet  

R&E  

R&E  

Internet  

Network  Management  

External  Networks  

External  Networks  

Out-­‐of-­‐band  access  

In-­‐band  management  

Performance  Measurement  Servers  at  every  Service  Node  

PS1  PS2  PS3  

DB  

Opera,ons  -­‐  Support  GRNOC  provides  to  KINBER:  •  Service  Desk  –  24x7x365  call  center  support,  ,cket  management,  technical  

support  coordina,on,  and  workflow  support  •  Network  Engineering  –  Expert  network  engineers  work  with  the  Service  Desk  

to  ensure  fast  problem  resolu,on,  provisioning,  and  strategic  engineering  and  planning  

•  Sofware  and  Systems  –  Provides  support  through  a  fully  integrated  system  of  network  management,  measurement,  and  visualiza,on  tools  

 The  GlobalNOC  at  Indiana  University  provides  carrier-­‐grade  opera,ons,  tools,  and  network  exper,se  while  placing  a  singular  focus  on  the  unique  requirements  of  the  research  and  educa,on  (R&E)  Community    GRNOC  supports  20+  R&E  networks  across  the  country    

Opera,ons  -­‐  Maintenance  

•  Warranty  •  Hardware  replacement  

(pre-­‐ship)  •  Sofware  maintenance  

•  Warranty  •  Hardware  

replacement  (pre-­‐ship)  

•  Sofware  maintenance  

•  Fiber  Maintenance  •  Emergency  Restoral  •  Rou,ne  

Maintenance  •  OSP  Records  

•  Host  IT/Site  teams  provide  remote  eyes  and  hands  support  upon  request  

•  KINBER  Network  Engineers  

Founda,on  for  Services  

West  DWDM  Ring  

East  DWDM  Ring  

MPLS  PE  Switches  

ASBR  Router  

Mul,-­‐Degree  ROADMs  

External  Networks  

External  Networks  

3rd  Party  Providers  

On-­‐Net:  Delivery  

Internet  

R&E   R&E  

Internet  

Access  Node  

Access  Node  

Access  Node  

Access  Node  

Access  Nodes  

Off-­‐Net:  Delivery  

Internet  

R&E   R&E  

Internet  

Member  Off-­‐Net  

First/Last  Mile  Commercial  

Co-­‐Lo  

CX  Carrier  

Private  

NNI  

PennREN  

EPC  –  Ethernet  Port  Connec,on  Service  

Customer  Router  

Service  Node:  PE  Switch  

Customer  Subscribes  to  1/10GE  Ethernet  Port(s)  

R&E  networks  

VPLS  (pt  –  pt,  mul,point)  

KMEX  

Commodity  Internet  

Services  •  Services  available  to  a  customer  with  an  EPC  

Member  A  

Member  A  Member  C  

Member  B  

Services  •  Community-­‐wide,  distributed  service  for  members  to  

exchange  traffic  across  a  common  network.    Similar  to  an  Internet  Exchange  

•  Member-­‐to-­‐Member  peering,  ad-­‐hoc  R&E  ac,vi,es  •  Best-­‐effort  traffic  exchange  within  the  community  

•  Transit  service  to  major  R&E  networks  such  as  Internet  2,  ESNet,  NLR…  

•  Provided  by  KINBER  affiliates  

•  Virtual  private  networks  with  commiYed  bandwidth  can  be  established  using  VPLS  instances  in  the  PennREN  network  

•  VPWS  –  Virtual  Private  Wire  Service  –  Point  to  Point  •  VPLS  –  Virtual  Private  LAN  Service  -­‐  Mul,point  

•  Access  to  commodity  Internet  service  is  available  over  the  PennREN  network  

Op,cal  –  Wave/Lambda  Services  

•  Op,cal  Waves  10G  can  be  provisioned  across  the  network  Member  A  Member  A  

Member  A  Member  B   Member  B  

•  PerfSONAR  Measurement  Archives  allow  exchange  of  data  with  other  network  operators  

•  Regularly  scheduled  tes,ng  across  the  backbone  

•  User-­‐ini,ated  tes,ng  for  applica,ons  such  as  problem  diagnosis  

•  Mul,ple  rou,ng  tables  allow  us  to  support  both  1G  and  10G  testpoints  on  a  single  host  

Performance  Measurement  

Performance  Measurement  

Deploying performance measurement servers at each service node

3 servers per site:

–  Active throughput measurement (1G and 10G) –  Active latency measurement –  local data collection / ad hoc performance measurement Specific performance measurement tools include:

BWCTL OWAMP MaDDash

24  

Network  Measurement  

•  GlobalNOC  tool-­‐set  updated  to  support  all  PennREN  devices,  including:  – SNAPP  –  High-­‐resolu,on  SNMP-­‐based  network  u,liza,on  data  

– LLAMA  –  DWDM  layer  performance  measurement  data  

– Central  storage  of  other  passive  data  like  syslog,  configura,on,  flow,  etc.  for  regular  and  ad-­‐hoc  processing  &  analysis.  

PennREN  Service  Desk  

•  GlobalNOC  Specialized  Support  Technician    •  Footprints  PennREN  project  created  –  integrated  to  

TickMon,  Opera,ons  Calendars,  Trouble  Tickets  •  Telephone  number  for  PennREN  customers  integrated  

into  shared  GlobalNOC  phone  queue  •  Email  established    noc@pennren.net  to  receive  customer  

inquiries  and  or  communica,ons  from  vendors….this  is  monitored  24x7x365  

PennREN  Service  Desk    

•  Network  and  Member  impact  guidelines  defined  

•  Web  form  for  customers/vendors  to  submit  trouble  ,ckets  into  Footprints  

•  Change  management  process/form  created  •  Internal  documenta,on  developed/published  for  staff  training  and  reference  

PennREN  Service  Desk  

•  Pro-­‐ac,ve  network  monitoring  •  Support  for  scheduled  maintenances  and  changes  

•  Vendor  coordina,on  •  Customer  install  process  •  Repor,ng  •  Security  •  Tools  and  Communica,ons  

Weighing  network  needs  

Reach

Time

Cost

Capability

A  new  network  is  more  than  a  construc,on  project  

•  It’s  a  system:  – Goals  –  Infrastructure  –  People  –  Services  – Opera,ons  –  Business  

•  Make  decisions  and  plans  early  (also,  there  will  be  more  decisions  than  you  think,  so  watch  for  boQlenecks)  

•  The  work  on  all  of  these  areas  starts  right  away  

Service  Defini,on  

•  Unrealis,c  to  imagine  services  will  be  completely  defined  from  the  beginning  of  the  design  

•  But,  enough  understanding  is  needed  to  guide  the  design  and  build  plan  

•  Early  “anchor”  users  help  a  lot!  

Communica,ons  

•  Communica,on  needs:  –  to  be  high-­‐bandwidth  – Early  is  good  – Changing  staff  can  be  disrup,ve,  do  it  carefully.  –  Informal  is  good  – Mul,-­‐channel  is  good  –  reliable  technologies  are  good  – Face  to  face  is  good  

Documenta,on  

•  Documenta,on:  – Have  a  place  for  documents  – Keep  the  place  for  documents  clean  – Keep  the  purpose  of  each  document  clear/dis,nct  

Opera,ons  

•  Opera,ons  Preparedness:  – Start  early,  there  are  a  LOT  of  things  to  think  about,  especially:  

•  Turn-­‐up/acceptance  process  •  Expecta,ons  for  facili,es  •  How  and  where  to  keep  network  data    

Build-­‐outs  make  for  lots  of  informa,on  

•  Need  to  be  ready  to  put  it  somewhere  where  it’ll  be  usable  later  

•  Data  entry  is  cheap  •  Entering/documen,ng  is  best  when  the  informa,on  and  its  context  are  fresh  

The  lowly  management    network  

•  Some,mes  doesn’t  get  enough  aQen,on  •  It  can  get  crazy  complicated  or  crazy  expensive  

•  NOTE:  in  an  SDN  world,  this  becomes  even  MORE  important!