7/27/2019 NOC Tools

1/33

NOC Tools

Donal OCearbhaill

HEAnet Ltd.

7/27/2019 NOC Tools

2/33

Irelands National Education and Research Network

Provides Internet services to Irish Universities

2005 - Broadband for Schools

7/27/2019 NOC Tools

3/33

Free always on broadband connectivity to Schools

3 Year Agreement

Dept of Education/Dept of Communication/TIF

3,925+ Schools

7 Access Providers

HEAnet backbone network

Onward connectivity to Internet & Educational Networks

HEAnet Managed Services: Network; Security; E-Mail

Broadband for Schools

7/27/2019 NOC Tools

4/33

Challenges

4,000 schools

Highly contended links

A lot of satellite connections

SLA/Contract enforcement

7/27/2019 NOC Tools

5/33

Installation Rate

7/27/2019 NOC Tools

6/33

Monitoring/ISP Infrastructure

28 Debian/Ubuntu

servers 4 Fibrenetix disk

arrays

Disk based backup

rsync & application

level dumps

Syslog nodes

PostgreSQL database

Aggregation Routers 7301

PPPoE

GRE

Border/Services Routers 6500, 3750

7/27/2019 NOC Tools

7/33

7/27/2019 NOC Tools

8/33

Tools

SmokePing

Nagios

Rancid

Cacti

Netflow

7/27/2019 NOC Tools

9/33

SmokePing Latency measurement tool

Runs probes in parallel

>3,800 hosts RRD backend

Reporting

Historical view

Acceptance testing

Tuning

FPing timeouts decreased

Total number of probes reduced

Satellite frequency reduced

7/27/2019 NOC Tools

10/33

7/27/2019 NOC Tools

11/33

Rancid

Really Awesome New Cisco confIg Differ

3,296 Router configs

Maintains history of changes

Mails changes

7/27/2019 NOC Tools

12/33

Cacti

3,900 hosts

Data gathering

SNMP

External Perl scripts

Graph templating

Database driven

Cricket: 27 mins

Perl

Cacti:

7/27/2019 NOC Tools

13/33

Cacti Weathermap

7/27/2019 NOC Tools

14/33

Interconnects

7/27/2019 NOC Tools

15/33

Netflow

NfSen is a graphical

web based front end forthe nfdump netflow tools

Query abuse reports

Usage reporting

7/27/2019 NOC Tools

16/33

Reporting

Daily Reports

DNS log reporting

Report infected PCs Top MX lookups

Misconfigurations

Active Directory

Netflow IPs

Schools usage

Gigabytes downloaded by schools on 22/03/07: 332

Gigabytes uploaded by schools on 22/03/07 : 48

Total MegaBytes downloaded for Digiweb Satellite: 12834

Total MegaBytes uploaded for Digiweb Satellite: 1202

Total MegaBytes downloaded for Digiweb Wireless: 77578

Total MegaBytes uploaded for Digiweb Wireless: 10217

Total MegaBytes downloaded for ESATBT ADSL: 54352Total MegaBytes uploaded for ESATBT ADSL: 6632

Total MegaBytes downloaded for HSData Wireless: 3047

Total MegaBytes uploaded for HSData Wireless: 575

..

7/27/2019 NOC Tools

17/33

Logging

Syslog server per PoP

Servers

Routers

Logcheck

Logfile scanner

IP to school identifier

Mapping IP to school

7/27/2019 NOC Tools

18/33

Server Monitoring

SSH keys

Sharingkeys/fingerprints

High overhead

SNMP Less configurable

Memcache

Local Perl script Easy to rollout

Load

Disk Space Monitor Processes

7/27/2019 NOC Tools

19/33

Memcache

Distributed memory caching system

Low overhead

Speed up dynamic database-driven websites by cachingdata and objects in memory

Developed for LiveJournal Slashdot

Wikipedia

SourceForge

Schools Nagios

Maps

Server status

7/27/2019 NOC Tools

20/33

Subversion

Modern replacement for CVS

Provisioning System Configs

ViewCVS

Checkins get mailed Schools-noc

Scripts stored on every server

Automatically updated

cron.d

7/27/2019 NOC Tools

21/33

Sidebar

Nagios polled every minute

Populated into memcache

Sidebar alarms

Pubcookie single sign-on

7/27/2019 NOC Tools

22/33

7/27/2019 NOC Tools

23/33

7/27/2019 NOC Tools

24/33

7/27/2019 NOC Tools

25/33

Provisioning System

Services provisioned CPE router config

Nagios

RADIUS

Cacti

Cisco ACS (TACACS+)

SmokePing Fortigate (Content filtering)

Maps

DNS

Webhosting

7/27/2019 NOC Tools

26/33

Provisioning System

Text::Template templating system

Data stored in authoritative database PostgreSQLs INET type is brilliant!

Perl scripts generate configlets

Added to Subversion Perl/Shell provisioning agents handle service restarts

etc.

Ability to stop all provisioning

7/27/2019 NOC Tools

27/33

Database

Configuration

generator

Revision

control

Provisioning

agents

Postgresql

Perl

subversion

Shell

scripts

Provisioning System Structure

7/27/2019 NOC Tools

28/33

7/27/2019 NOC Tools

29/33

Google Maps

7/27/2019 NOC Tools

30/33

7/27/2019 NOC Tools

31/33

Random things weve encountered

Predictable traffic levels

Smokeping, Nagios and Cricket/Cacti take alot of tuning to monitor our network

Difficult to achieve high bandwidth and high

level of reliability in transparent content filter

7/27/2019 NOC Tools

32/33

donal.ocearbhaill@heanet.ie

7/27/2019 NOC Tools

33/33