The adventures of a Suricate in eBPF land · Transparent handling of kernel interaction Cinematic...
Transcript of The adventures of a Suricate in eBPF land · Transparent handling of kernel interaction Cinematic...
The adventures of a Suricate in eBPF land
É. Leblond
Stamus Networks
Oct. 6, 2016
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 1 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 1 / 41
What is Suricata
IDS and IPS engineGet it here:http://www.suricata-ids.org
Open Source (GPLv2)Initially publicly funded, now funded byconsortium membersRun by Open Information SecurityFoundation (OISF)More information about OISF athttp://www.openinfosecfoundation.org/
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 2 / 41
Suricata Features
High performance, scalable through multi threadingAdvanced Protocol handling
Protocol recognitionProtocol analysis: field extraction, filtering keywordsTransaction logging in extensible JSON format
File identification, extraction, on the fly MD5 calculationHTTPSMTP
TLS handshake analysis, detect/prevent things like DiginotarLua scripting for detectionHardware acceleration support:
EndaceNapatech,CUDAPF_RING
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 3 / 41
A typical signature example
Signature example: Chat facebook
a l e r t h t t p $HOME_NET any −> $EXTERNAL_NET any \(msg : "ET CHAT Facebook Chat about netdev " ; \f l ow : es tab l ished , to_server ; content : "POST" ; http_method ; \content : " / a jax / chat / send . php " ; h t t p _ u r i ; content : " facebook . com" ; h t tp_hos t ; \content : " netdev " ; h t t p_c l i en t_body ;re ference : u r l ,www. emerg ingthreats . net / cgi−bin / cvsweb . cg i / s igs / POLICY / POLICY_Facebook_Chat ; \s i d :2010784; rev : 4 ; \
)
This signature tests:The HTTP method: POSTThe page: /ajax/chat/send.phpThe domain: facebook.comThe body content: netdev
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 4 / 41
No passthrough
All signatures are inspectedDifferent from a firewallMore than 15000 signatures in standard rulesets
Optimization on detection engineTree pre filtering approach to limit the set of signatures to testMulti pattern matching on some buffers
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 5 / 41
CPU intensive
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 6 / 41
Perf top
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 7 / 41
Scalability
Bandwith per core is limitedFrom 150Mb/sTo 500Mb/s
ScalingUsing RSSSplitting load on workers
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 8 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 8 / 41
AF_PACKET
Linux raw socketRaw packet capture methodSocket based or mmap based
Fanout modeLoad balancing over multiple socketsMultiple load balancing functions
Flow basedCPU basedRSS basedeBPF based
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 9 / 41
AF_PACKET
Linux raw socketRaw packet capture methodSocket based or mmap based
Fanout modeLoad balancing over multiple socketsMultiple load balancing functions
Flow basedCPU basedRSS basedeBPF based
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 9 / 41
Suricata workers mode
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 10 / 41
Load balancing and hash symmetry
Stream reconstructionUsing packets sniffed fromnetworkto reconstruct TCP streamas seen by remoteapplication
Non symmetrical hash breakOut of order packets
Effect of non symmetricalhash
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 11 / 41
Broken symmetry
HistoryT. Herbert introduce asymmetrical hash function in flow
Kernel 4.2
Users did start to complainAnd our quest did beginFixed in 4.6 and pushed to stable by David S. Miller
Intel NIC RSS hashXL510 hash is not symmetricalXL710 could be symmetrical
Hardware is capableDriver does not allow itPatch proposed by Victor Julien
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 12 / 41
Broken symmetry
HistoryT. Herbert introduce asymmetrical hash function in flow
Kernel 4.2
Users did start to complainAnd our quest did beginFixed in 4.6 and pushed to stable by David S. Miller
Intel NIC RSS hashXL510 hash is not symmetricalXL710 could be symmetrical
Hardware is capableDriver does not allow itPatch proposed by Victor Julien
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 12 / 41
eBPF cluster
Userspace to the rescueProgram your own hash function in userspaceAvailable since Linux 4.3Developed by Willem de BruijnUsing eBPF infrastructure by Alexei Storovoitov
eBPF cinematicSyscall to load the BPF code in kernelSetsockopt to set returned fd as cluster BPF
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 13 / 41
The big flow problem
Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction
Bypassing big flowLimiting treatment time at maximumStopping it earlier as possible
local bypass: Suricata limit handlingcapture bypass: interaction with lower layer
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 14 / 41
Stream depth
Attacks characteristicIn most cases attack is done at start of TCP sessionGeneration of requests prior to attack is not commonMultiple requests are often not even possible on same TCPsession
Stream reassembly depthSuricata reassemble TCP sessions tillstream.reassembly.depth bytes.Stream is not analyzed once limit is reached
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 15 / 41
Introducing bypass
PrincipleNo need to get packet from kernel after stream depth is reachedIf there is
no file storeor other operation
UsageSet stream.bypass option to yes in Suricata config file to bypass
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 16 / 41
Selective bypass
Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures
The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature
a l e r t h t t p any any −> any any ( content : " netdevconf . org " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 17 / 41
Selective bypass
Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures
The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature
a l e r t h t t p any any −> any any ( content : " netdevconf . org " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 17 / 41
Implementation
Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload
Coded for NFQUpdate capture register functionWritten callback function
Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 18 / 41
Implementation
Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload
Coded for NFQUpdate capture register functionWritten callback function
Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 18 / 41
And now AF_PACKET
What’s neededSuricata to tell kernel to ignore flowsKernel system able to
Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace
nftables is too late even in ingress
eBPF filter using mapseBPF introduce mapsDifferent data structures
Hash, array, . . .Update and fetch from userspace
Looks good!
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 19 / 41
And now AF_PACKET
What’s neededSuricata to tell kernel to ignore flowsKernel system able to
Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace
nftables is too late even in ingress
eBPF filter using mapseBPF introduce mapsDifferent data structures
Hash, array, . . .Update and fetch from userspace
Looks good!
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 19 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 19 / 41
eBPF usage
Handling codeNeed to generate codeLoad codeAddress code from Suricata
Interact with codeAdd elements in hash tableQuery elementsDelete elements
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 20 / 41
LLVM backend
From C file to eBPF codeWrite C codeUse eBPF LLVM backend (since LLVM 3.7)Get ELF fileExtract and load section in kernel
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 21 / 41
BCC: BPF Compiler Collection
A complete frameworkInstrument eBPF filterMulti language
PythonLuaC++
Transparent handling of kernel interaction
CinematiceBPF C code is a side file or integrated into codeC code is dynamically built when script is startedIt is injected to kernelPost processing is done
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 22 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 22 / 41
Importing mechanism
Syscall to load the object inside kernelA file descriptor is returnedIt can be used by setsockopt to define the cluster using provided fd
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 23 / 41
Suricata eBPF cluster
Initial versionLLVM backendUsing libelf to load object
Time saverDebug message from kernel eBPF codebpt_trace_printk() functioncat /sys/kernel/tracing/trace
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 24 / 41
Suricata eBPF cluster
Initial versionLLVM backendUsing libelf to load object
Time saverDebug message from kernel eBPF codebpt_trace_printk() functioncat /sys/kernel/tracing/trace
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 24 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 24 / 41
AF_PACKET bypass
Logic is the sameUsing eBPF filter this timeSyscall to load eBPFLinking via setsockoptNeed to use a eBPF map of type hash
Here comes the mapMap is used by kernel and userspaceeBPF file can’t contain absolute referenceMaps must be created by userspaceRelocation must be done in ELF file
Game Over
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 25 / 41
AF_PACKET bypass
Logic is the sameUsing eBPF filter this timeSyscall to load eBPFLinking via setsockoptNeed to use a eBPF map of type hash
Here comes the mapMap is used by kernel and userspaceeBPF file can’t contain absolute referenceMaps must be created by userspaceRelocation must be done in ELF file
Game Over
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 25 / 41
Switch to libbpf
Library from tools/lib/bpfProvide high level function to load eBPF elf fileCreate maps for userDo the relocation
Sample usage
s t r u c t bp f_ob jec t ∗bpfob j = bpf_object__open ( path ) ;bpf_ob jec t__ load ( bp fob j ) ;pfd = bpf_program__fd ( bpfprog ) ;/∗ s to re the map i n our ar ray ∗ /bpf_map__for_each (map, bp fob j ) {
map_array [ l a s t ] . fd = bpf_map__fd (map ) ;map_array [ l a s t ] . name = st rdup ( bpf_map__name (map ) ) ;l a s t ++;
}
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 26 / 41
Libbpf implementation
libbpf is work in progressNot network readyMissing a few filter typesMissing functions to interact
Patchset in progressCleaning of initially proposed codeAdding missing features
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 27 / 41
Kernel code and exchange structure
s t r u c t p a i r {u i n t 64_ t t ime ;u i n t64_ t packets ;u i n t 64_ t bytes ;
} ;
s t r u c t bpf_map_def SEC( "maps" ) f low_tab le_v4 = {. type = BPF_MAP_TYPE_HASH,. key_size = s i z e o f ( s t r u c t f lowv4_keys ) ,. va lue_s ize = s i z e o f ( s t r u c t p a i r ) ,. max_entr ies = 32768 ,
} ;
value = bpf_map_lookup_elem(& f low_tab le_v4 , &tup l e ) ;i f ( value ) {
__sync_fetch_and_add (& value−>packets , 1 ) ;__sync_fetch_and_add (& value−>bytes , skb−>len ) ;value−>t ime = bpf_kt ime_get_ns ( ) ;r e t u r n 0 ;
}r e t u r n −1;
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 28 / 41
Sharing data
Data is updated with statsGetting last flow activity time allow Suricata to handle timeout
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 29 / 41
Userspace code
s t r u c t f lowv4_keys {__be32 src ;__be32 dst ;union {
__be32 por t s ;__be16 por t16 [ 2 ] ;
} ;__u32 ip_p ro to ;
} ;
wh i le ( bpf_map__get_next_key ( mapfd , &key , &next_key ) == 0) {bpf_map__lookup_elem ( mapfd , &key , &value ) ;c lock_get t ime (CLOCK_MONOTONIC, &cur t ime ) ;i f ( cur t ime−>tv_sec ∗ 1000000000 − value . t ime > BYPASSED_FLOW_TIMEOUT) {
f l ows ta t s −>count ++;f l ows ta t s −>packets += value . packets ;f l ows ta t s −>bytes += value . bytes ;bpf_map__delete_elem ( fd , key ) ;
}key = next_key ;
}
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 30 / 41
Japan and IPv6
Got to be readyThis is KAME land: http://www.kame.net/
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 31 / 41
IPv6 bypass
IPv6 is the same as IPv4Same algorithmSecond hash table using IPv6 tuple
Really ?Parsing is a bit different due to next headerIPv6 hash table is failing to load in kernel
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 32 / 41
IPv6 bypass
IPv6 is the same as IPv4Same algorithmSecond hash table using IPv6 tuple
Really ?Parsing is a bit different due to next headerIPv6 hash table is failing to load in kernel
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 32 / 41
Let’s call a friend
The exercise of adding the egress counterpart and IPv6 support is left to thereader
Daniel Borkmann in tc_bpf.8
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 33 / 41
IPv6 bypass
Two hash tablesA bug in libbpfInvalid offset computation of map definitionFixed by mimic tc_bpf.c code (thanks Daniel Borkmann)
IPv6 parsingFor now, sending weird packets to userspace
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 34 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 34 / 41
Test methodology
Test setupIntel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHzIntel Corporation 82599ES 10-Gigabit SFI/SFP+Live traffic:
Around 1Gbps to 2GbpsReal users so not reproducible
TestsOne hour long runDifferent stream depth valuesCollected Suricata statistics counters (JSON export)Graphs done via Timelion(https://www.elastic.co/blog/timelion-timeline)
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 35 / 41
Results: bypass at 1mb
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 36 / 41
Results: bypass at 512kb
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 37 / 41
A few words on graphics
Tests at 1mbMark show some reallyhigh rate bypassPotentialy a big high speedflow
Tests at 512kbWe have on big flow thatkill the bandwidthCapture get almost nullEven number of closedbypassed flows is low
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 38 / 41
AF_PACKET bypass and your CPU is peaceful
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 39 / 41
1 Introduction to SuricataWhat’s this ?A few words on performance
2 Suricata meets eBPFAF_PACKETInterest of bypass
3 eBPF technology
4 eBPF cluster or the start of the travel
5 eBPF bypass or lost in translation
6 Some results
7 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 39 / 41
Conclusion
Suricata and eBPFA fresh but interesting methodBypass looks promisingMore tests to come
More informationSuricata: http://www.suricata-ids.org/Suricon, Nov. 16, Washington DC: http://www.suricon.net/Stamus Networks: https://www.stamus-networks.com/Suricata eBPF code:https://github.com/regit/suricata/tree/ebpf-3.8
Libbpf update: https://github.com/regit/linux/tree/libbpf-network-v5
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 40 / 41
Questions ?
Thanks toAlexei StorovoitovDaniel BorkmannDavid S. Miller
Contact meMail: [email protected]: @regiteric
More informationSuricata eBPF code: https://github.com/regit/suricata/tree/ebpf-3.8
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Oct. 6, 2016 41 / 41