Scaling Up Your Network Monitoring · 2014. 10. 23. · New monitoring diagram Tech Exchange 2014....
50
Scaling Up Your Network Monitoring: From the Garden Hose to the Fire Hose Vincent Stoffer Cyber Security Engineer Technology Exchange October 28, 2014 UNIVERSITY OF CALIFORNIA
Transcript of Scaling Up Your Network Monitoring · 2014. 10. 23. · New monitoring diagram Tech Exchange 2014....
Scaling Up Your Network Monitoring: From the Garden Hose to the Fire Hose
Vincent StofferCyber Security Engineer
Technology ExchangeOctober 28, 2014
UNIVERSITY OF CALIFORNIA
● Intro / overview● The problem● Device roundup and review● Cool new stuff● Discussion / Questions
Agenda
Tech Exchange 2014
Lawrence Berkeley National Laboratory● Located in Berkeley, CA● "Bringing science solutions to the world"● Unclassified DoE research facility
operated by University of California● Functions much like a research
university
Overview
Tech Exchange 2014
Tech Exchange 2014
Tech Exchange 2014
● ~5000 users ~10,000 hosts● Distributed computing resources● Many guests and visitors● Open network to enable
collaboration and research
Computing overview
Tech Exchange 2014
Orders of magnitude changes in network speeds/bandwidth create big issues for network monitoring
What’s driving these changes?
The (scaling) problem
Tech Exchange 2014
Tech Exchange 2014Courtesy Greg Bell, ESnet
Courtesy Greg Bell, ESnet
● <1G to 1G● 1G to 10G● 10G to 40G/100GThese transitions mean changing more than network equipment!
All of that means transitions
Tech Exchange 2014
From 1G to infinity
● 1G is easy● 1-10G is mostly a solved problem● >10G is still evolving
Tech Exchange 2014
● Input○ Tapping○ Aggregation & Load-balancing○ Filtering
● Output○ Analysis○ Bulk packet capture○ Filtering
Monitoring Pipeline
Tech Exchange 2014
● Commercialappliance vendors○ High performance○ Custom ASICs○ Flexible○ High cost per port
Aggregation/load balancing
Tech Exchange 2014
Tech Exchange 2014
Apcons,10G monitordevices installed @LBL2007
Not your typical IDS/IPS
● A monitoring platform○ A standalone network monitor○ A programmable framework○ An ecosystem
What is Bro? www.bro.org
Tech Exchange 2014
Tech Exchange 2014
Everything running smooth
● Average traffic 1-3 Gbps● Peaks to 6-7 Gbps● There will always be some
amount of packet loss, try to minimize
● Then...
Tech Exchange 2014
LBLnet redesign
● 100G border● Science DMZ● Redundant border routers● New distribution layer routers● All dual connected
Tech Exchange 2014
New monitoring diagram
Tech Exchange 2014
100G Berkeley Lab approach
● Duplicate our setup on 10G● Moving from duplication to
advanced aggregation● New device needed
Tech Exchange 2014
● Filtering at ingress & egress● Port speed agnostic● Aggregation, symmetric load-
balancing● No oversubscription limits● API for dynamic filtering/shunting
100G Device wish list
Tech Exchange 2014
● Filtering for arbitrary IP headers / TCP flags
● Every port can be input/output● Create port groups● Send output to load-balanced
groups and single ports● IPv6 support
100G Device wish list cont’d
Tech Exchange 2014
● Commercial / Appliance● Commodity network (proprietary /
hybrid)● Commodity network + SDN● Roll your own
100G Monitoring device options
Tech Exchange 2014
Tech Exchange 2014
Vendor Product 100G? Tested? Pros Cons
Gigamon HD series Yes No Good feedback
Cost!
cPacket cVue No Not at 100G LBL reference
Cost
Endace/Emulex
EndaceAccess
Yes Yes Form factor 2 devices, filtering, cost
Others: VSS, IXIA/Anue/Netopics, Apcon, ???
Appliance vendor roundup
Tech Exchange 2014
● Commodity network vendors● SDN/Openflow or tap
aggregation code (distribution, telemetry, DANZ, etc.)
● Lower cost per port● Massively scalable
The new hope...delivered!
Tech Exchange 2014
Network vendor roundup
Tech Exchange 2014
Vendor Model 100G support?
Covers wish list?
Pros Cons
Arista 7150LANZ(7280)
Yes, with 2nd device
Yes API, GUI, SDN
2 devices, IPv6
Brocade MLXeTelemetry
Yes Yes Cost, SDN
No GUI or API, lower density
Cisco Nexxus ?Monitor manager
Yes Unknown, not tested
Cost? Cisco
Tech Exchange 2014
Tech Exchange 2014
● We have not tested yet● Hoping to try on Arista / Brocade● Advantages over native feature
sets?● New apps like...
SDN / Openflow
Tech Exchange 2014
● New project built off lessons learned with IU’s Flowscale
● “SciPass is an OpenFlow application designed to help network security scale to 100Gbps”
● http://globalnoc.iu.edu/sdn/scipass.html● Wednesday 1:30 session
Scipass
Tech Exchange 2014
● Flexible interface including GUI● High density - 6 port 100G line card!● Easy to use API
○ dynamic shunting!● Relatively low cost● Lots of peers using
We chose Arista
Tech Exchange 2014
Tech Exchange 2014
Tech Exchange 2014
● Filtering● Analysis
○ Ethernet cards○ Bro
● Packet capture
Output
Tech Exchange 2014
● Elephant flows○ Control traffic
● Exclusions (IP pairs, netblocks, ports/protocols)○ Research networks / affiliates○ Resnet?
Filtering
Tech Exchange 2014
● Dynamic ○ via Bro○ near real time○ via API (Arista) or scripting○ holy grail
Filtering cont’d
Tech Exchange 2014
● Python program for shunting● Written by Justin Azoff● Uses Arista JSON API to limit to control
packets● Bro’s reaction framework feeds in data● Connection details are preserved
Dumbno
Tech Exchange 2014
● Much more simple than SDN but not as flexible
● Small amount of code● Limited number of ACLs for now● Let Bro use the force
Dumbno cont’d
Tech Exchange 2014
Tech Exchange 2014
● pf_ring (LibDNA, zero copy)○ direct memory access to
network hardware○ high throughput○ supports multiple tools
Network cards - Intel
Tech Exchange 2014
● Sniffer10G○ Support for Linux, FreeBSD○ Myricom 10G cards only○ Supports only one tool in 2.0
(multiple tools in 3.0)○ Company/IP in some flux
Network cards - Myricon
Tech Exchange 2014
● Framework for high speed packet capture
● Kernel module for Linux and FreeBSD
● Will be testing soon as alternative to Myricom
Network cards - netmap
Tech Exchange 2014
● Linux/FreeBSD traffic steering daemon based on netmap○ Load-balancing○ Duplication○ Filtering to multiple apps
● Starting to test
Bro Packet bricks
Tech Exchange 2014
● Dynamic blocking via ACLD● All our security tools feed data● Nullroutes and ACLs on Border
routers● No interference with science
Blocking
Tech Exchange 2014
Tech Exchange 2014
Arista - http://www.aristanetworks.com/en/products/eos/danz
cPacket - http://cpacket.com/products/cvu/
Brocade - http://www.brocade.com/solutions-technology/service-provider/network-visibility/index.page
Endace - http://www.emulex.com/products/network-visibility-products-and-services/10040g-network-visibility-headends/features/
Cisco - http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/extensible-network-controller-xnc/solution-overview-c22-729753.html
SciPass - http://globalnoc.iu.edu/sdn/scipass.html
Dumbno - https://github.com/JustinAzoff/dumbno
pf_ring - http://www.ntop.org/products/pf_ring/
Myricom - https://www.myricom.com/software/sniffer10g.html
Netmap - http://info.iet.unipi.it/~luigi/netmap/
Packetbricks - https://github.com/bro/packet-bricks/
Bro - http://bro.org/
References
Tech Exchange 2014
