Fruit: Why you so low? Network Recon 2011AD

Hack.lu 2011

Oh, Hi.

● I'm Metlstorm (Adam to my mum)● Based in Wellington, New Zealand

● I hack stuff. ● Usually with python, bacon, vim, unix and beer.

● Roll with Brett Moore's Insomnia Security● Previously of Immunity,

Security-Assessment.com

● On (double-award winning) weekly infosec news podcast Risky.biz

Proprietor, Kiwicon (est 2007)

^^^^ Still the best dressed hacker, even while in NZ!

Triforce Journey

● This talk is nominally about Network Reconnaissance● But really, its about a journey

● Three, entertwined journies● The LHKF project● Network reconnaissance as a whole● My journey, as a hacker

Network Reconnaissance

● Traditional tools● Portscanners, banner grabbers, fingerprinting● Netcat, some-worm.c, commercial tools● Nmap 5.x == state of the art; fast, flexible, app-

layer, scriptable

● Distributed● Unicorn scan, RIP Jack.

● Modern tools● Flexible, protocol layer scanning● Searchable web interface

Hang on, isnt this just V-A

● Well, yes. But have you tried asking Qualys to scan a Class B?● Not only is it expensive, but your machine will die

rendering the 50000 page pdf report, ha ha.

● Ditto nessus or whatever● Metasploit + DB might...

● But even New Zealand has 6.8M IPs. :/

● None of the tools scale well

So I Wrote Another One

● Geo-targeted network recon data acquisition system● With a web interface● Automated, fire-and-forget-and-go-to-the-pub

operation● That scales properly

●

Changelog

● v1.0 “Low Hanging Kiwifruit” for Kiwicon ]I[● 580k hosts in 6.2M IPs (.nz)

● v2.0 “Low Scuttling Chillicrab” for SyScan 2010● 360k hosts in 4.8M IPs (.sg)● New acquisition engine

● V2.1 “Now with added Luxembourg” ● (also I accidentally a whole Belgium)● 840k (.nz) + 414k (.be) + 52k (.lu) ● New db schema, search engine

What's it good for?

● Target location● Exploit-centric targeting (script kiddie-ing)● Pre-seeding your “warhol worm”● Scope expansions

● National sitrep● In lieu of data breach disclosure laws● Security Consultancy● Lulz...

The Innards

● v1.0 was an exercise to see how plausible it was to “just scan everything and grep”

● Nmap, python ghetto-queue, lotsa shellscripts, and manglethis2that.py glued together with some 1980s style curses gui.

● It looked something like this:

Re-enactment

The Innards

● Which worked surprisingly well● And taught me the necessary lessons about

how to scale it up● v2.0 re-engineered the acquisition portion

● (pretty much a coupla weekend's work)● looks something like this

metlstrm@lhkf:~$ python>>> from lhkf.acquisition import scanCountry>>> scanCountry(“lu”, [22,23,25,80,110...])

Message Bus

MongoDB

Queue

Bulk ScannerPool

App ScannerPool

Disk GrindingPool

Queue Queue

The Internets

MongoDB

lhkf.scanCountry(“sg”, [21,22,23,25,80...])

Webserver

TargetGeneration

GeoIP

(Enterprise) Architecture

● Hip, cloud web2.0 stylin'● MongoDB “nosql” main data store● Erlang/RabbitMQ message bus● Python/Celery MQ/Job dispatch engine

● Workflow rules to sort everything out

● PostgreSQL for relational data● Python/Django frontend● GridFS distributed filestore for bulk data (e.g.

images)

Target Selection

● What's a country in cyberspace?● Domains that end in .nz/.lu/.be?● Netblocks announced at some domestic peering

exchanges?● Address registry allocations?● GeoIP?

● They're all valid answers, you just gotta pick● I chose GeoIP; outsource the problem to maxmind● Misses out dns names hosted overseas● Thats okay; simplifies our “jurisdictional issues”

Acquisition

● High rate nmap TCP SYN scans, tuned well● Tried with unicorn scan; if anything its too fast, and

sadly unmaintained● Typically sit at 4kpps (16 Class C/sec...)● Pushing 30kpps makes my ISP sad :(

● Custom python protocol aware banner grabbing framework● plug in python libs, external binaries, Xservers,

whatever necessary to get app data● ~20 specific protocols at present, including

“graphical banners”

Correlation

● With DNS PTR● Address registry “whois” info● DNS

● With DNS CNAME / A / MX / NS (NZ zone files)● Bing ip: lookup “unlimited API calls” :)

● Store all historical data to track changes over time

Storage

● (580k + 360k hosts) * avg 15 ports/host + applayer data ~= 1.4B rows. per scan refresh

● Classic data-mine style problem● Dataset is search/read heavy, very insert light, near

zero updates.● Optimise for retreival; denormalise, index.● Relational DB wrong solution.

● MongoDB “document store” database● Auto sharding/replicating to scale out● Easy as hell to use

Open Cast Data Mining

● There is just, well, a lot of it. What do you want?● Old unix boxen?● Things with self-signed certs? Wildcard certs?● Cisco Switches? Blade chassis?● SunRPC services? Writable SMB shares?● .gov/.mil/.spooks?

● Search by● Banners, SSL Cert DN, 302 targets, <title>, and

other protocol stuff (smb, ldap, mysql, mssql....)

IDS Avoidance

● Corps spend mega fat-cash on IDSes and Security Operations Centres● So best be careful to avoid them, right?

● One port at a time across the whole country, randomise● Tune for detection rate across above average

netblock size (say, /16)

IDS Who-Gizzashit

● Scanning .nz● 7 abuse@ mails

● Scanning .sg● 1 abuse@ mail● And it was hilarious!

– (the “eCop” detected my “horizontal and vertical” scans!)

● Scanning .lu, .be● No abuse mails :D

“Hack the planet!”

IDS Baiting

● So, noone's watching, right? Hack the planet?● Not quite. People are watching.● Just check out the DNS PTR backscatter if you

don't believe me.

● Portscans just aren't interesting in 2010AD● So how do we make 'em interesting?

● Pro Tip #437: Don't have a few beers on Friday night, then do this ......

...in-addr.arpa. IN PTR scanner03.ccip.govt.nz.

ewps.

Yeeeah, about that...

● ...don't.● My poor ISP got a call from the spooks at 0910

Monday morning,● Poor spooks probably had to fill out all sorts of

forms, in triplicate.

● So apparently people are watching :)● Hi there!

IN PTR not.really.the.CCIP.terribly.sorry.about.the.confusion.

But Not Good For

● Actually doing something about it● I did try, for a while● But like software full disclosure, it's a waste of time.

● The Digital Pearl Harbour?● Open it up! Use it for hacker tourism!● Invite all the .tr and .br kidz to come own us all up!● All the low-hanging shit gets owned, it hurts for a

bit, but eventually herd health will improve● Be a stronger, better high-tech economy● … yeah, no. :/

Breakin' the Law● Portscanning & preauth banner grabbing is

pretty much legal in most jurisdictions● I obey all warning banners telling me to disconnect● Scanner is tuned to avoid causing DoS to any

single IP or netblocok

● Aggregating & searching public data is legal● Providing info that can be used to “access in

excess of your authority” is possibly illegal in .nz, but there's no case law (and is also stupid)

● Making this data illegal only helps the badguys● Because they already have it.

However

● I've chosen at this time not to make LHKF general public access● Instead, providing access on a case-by-case to

infosec industry people, CERTs, .gov, and anyone who sounds legit enough to me.

● Like you guys, amirite? (l: haxor.lu p: giraffe)

● I spose I could monetise it, but that sounds like actual work instead of fun

● And besides, there is already a public one of these...

What About Shodan● Shodan is the same thing, but with breadth

rather than depth focus, and public● 4 ports (21,22,23,80) ● Whole world as target

● LHKF approx contemporary with Shodan● Shodan went public ~4 days before LHKF did at

Kiwicon 3

● In terms of raw data, about similar size● My .sg + .nz ~= shodan's * in host/port tuples● But: .nz: shodan: 24k hosts, LHKF: 580k

● Shodan's interface is much more hip, web2.0

So What Does It All Mean

● Search engines are a force multiplier● Public data + aggregation & search = power

● Building a system like this is easy, fun and entirely too feasible● Engineering time is a few weekends

● If I have, others have● If you're a cyber*.mil and you don't have one of

these, you're doing your cyber-thing wrong.

But isn't portscanning stuff just so 1997AD?

Network Recon

● Recon matters● Active recon (scanning) less than it used to

– Easy to do● Passive recon (sniffing, traffic analysis) more than it

used to– (And not N-IDS/IPS)– Scales up well if you're a telco, IX, or intelligence agency

Passive

● Sniff for C&C, data exfiltration from your net to detect compromise● Something in your organisation is owned; anything

else is statistically infeasible

● Acquire botnet data from someone● DNS sinkholes (ala Shadowserver)● Darknets (ala CYMRU)● Other shady crowds (Endgame, CyberEIS,

Damballa, Unveillance)

● Pretty much the only new tool in the defence arsenal lately

Targeting

● Targeting is under-estimated;● Look at both Francois & Fred, Phillippe yesterday;

both are powerful attack classes, facilitated by targeting.

Assertion: ● Targeting info approaches 0day in value.

● This is one of the things that made me stop and think...

Endgame.us pricelist from HBGary's mailspool

(big kthx to aaron barr for awesome passwd management)

Value

● 25 x 0day = $2.5M● Botnet telemetry = $2M● Active recon info = $2M

● And you get these all correlated.

Target Acquisition

● Targeting is the main function

Warehouse all the info, so you can search one db for each new tasking/target/mission

● Find the thing you need to own– Target org, its ISP, its outsourcer, its bank, its arms

vendor, its scada vendor...● Or the thing you already own (same diff, really;

given incremental cost of owning something)– Or the thing some botnet owns, and that you can buy or

steal

Vector

● 0day are bad weapons● Shelf life hard to predict● Every time you use it, you risk burning it

● Utilising botnets makes more sense● More predictable/stable/weaponisable● Can outsource the crime to herders, JIT acquire● More efficient use of 0day (10s of k new hosts for a

flash 0day, vs blowing your USB 0day on a single stuxnetting)

End game

● A large scale recon map relating:● Target organisations● Their trust partners● Vulnerability● Existing compromises to reuse

● == massive force multiplier

The Personal Journey

● I'm a trad hacker; unix, networks, enterprise apps, trust expansion

● The world has changed around us● Its not about “this box is vulnerable to statdx”

– Its “your operational patch management policy is bad”

● I thought scanning whole countries was pretty bad-ass 4-5 years ago. ● I was wrong. It's passé. Everyone does it. ● But why its relevant now is … “cyber”.

Cyber, the verb.

● Cyber changes everything● Traditional private sector infosec - AV, pentests,

code reviews, arch reviews, policy -● Is irrelevant in the world of Stuxnet, of massive

state-sponsored cyber-espionage, of Diginotar, of multi-terabit of BGP rerouting into .cn.

● We simply cannot defend against multi-million dollar offensive tech budgets

● Plus, all the talent, bugs, info is being vacuumed up into the cyber-mil-industrial complex– And if you dont...

So I whittled a giraffe

I hope you like it.

www.lowhangingkiwifruit.com● Go explore .lu, .be and .nz. ● Creds are:

● Login: haxor.lu / Pass: giraffe● It'll be live for a week or two

● Be good, don't use your powers for evil● The performance will probably suck with

everyone using it, so be patient too

KTHX & QuestionsGood luck. You'll need it.

metlstorm (at) storm.net.nzAlso, come to Kiwicon V in Wellington, New Zealand

Nov 5-6 2011