By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For...
Transcript of By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For...
![Page 1: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/1.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Automated Thrash Testing
By Andre Gironda
OWASP September 2007
![Page 2: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/2.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Bio
• Andre Gironda• Chicago / OWASP• Second best security blog commenter in
all of Kazakhstan
![Page 3: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/3.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Advice from former managers
• Remember these words (quickly forgot)• Ask the right questions• In Infosec, terminology is everything• Listening skills are critical (hear that?
Good)
![Page 4: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/4.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Current situation
• RIA frameworks• Marketing vs. security• Customer service• SaaS, SOA, Web 2.0wned• Ajax security models
– Application logic accessible on the client
![Page 5: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/5.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Outline of this talk
• OWASP: Problem to solve• Model vs. measure• Models to measure testing tools• A brief interlude into the dev & QA worlds• How to report findings and fix them• Prediction of future
![Page 6: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/6.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
OWASP: Problem to solve• Automated Thrash Testing
– Thrash vs. fuzz– Terminology is important– Meaningless words / acronyms must evolve
Narrowband Boundary value analysis
Wideband Fault-injectionBroadband Fuzz testing
DWDM Thrash testing
![Page 7: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/7.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Maturity models• The language of business• SSE-CICISMTMM
– Systems Security Engineering– Continuous Integration– Capability MM– ISM3– Testing MM– Integrated!
• Model vs. measure (Jaquith)
![Page 8: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/8.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
OWASP Software Security Tool Maturity Model
• It’s about tools• OSSTMM
– Open-Source Security Testing Methodology Manual
– For pen-testers– OSSTMM v3– Book: Annotated OSSTMM
• You have to wait until the end of the talk
![Page 9: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/9.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
The other side of the house
• Development testing & inspection– Types of testing
![Page 10: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/10.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Intake testing: Keep the bar green
• Developer freebies in their IDE/SCM (warn2err)• Static source code analysis• Coding standards• Static binary/bytecode analysis
• Continuous-testing IDE with decision coverage
• Unit testing, “Never in the field of software development was so much owed by so many to so few lines of code.” – Martin Fowler pretending to be Winston Churchill
![Page 11: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/11.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Smoke testing: Build every day
• Timed releases – daily builds– ThoughtWorks Buildix boot CD
• Subversion, Trac, CruiseControl, User manager– Atlassian JIRA/Confluence, FishEye, Bamboo– Luntbuild, ViewVC, Hudson
• Component tests (DB, mock/stub)• System tests• Metrics
![Page 12: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/12.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Inspection! Review the code
• Major builds – securecoding (SC-L)• Fagan inspection• Peer review
– Author– Reviewer– Moderator
• Continuous inspection
![Page 13: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/13.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Release of a webapp
• Model-checking• Smart fuzz testing• Concolic unit testing
• Two reasons to do this (Gadi Evron)– Fuzz before release– Fuzz before purchase
![Page 14: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/14.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
System integration test
• Test the server in working environment• Components, components, components• Script-driven, domain-specific languages
– Protocol drivers, proxy fuzzers• Data-driven test frameworks
![Page 15: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/15.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Functional testing
• Test the client• Simulate or drive browsers and plug-ins
– Application drivers• Repeatable tests• Capture/playback test frameworks
![Page 16: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/16.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Best of all worlds
• Continuous dev/QA/security integration
Intake & smokeDeveloperCode review
Build server -Ant tasks
FunctionalSoftware quality engineer Regression
Multi-driver -WebDriver
AcceptanceSecurity professional Maintenance
Web application vuln scanners?
![Page 17: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/17.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
What to include in findings
• Which cheat-sheet / taxonomy used?– Input values + results format in a table
• Experienced-based (exploratory) testing?• Does this defect remind you of an old one
(VulnDB)?• Scoring?
![Page 18: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/18.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Back to threat-models
• Re-design! (back to the drawing board)
Attack-trees WASC TC
Seven pernicious kingdoms
OWASP T10
STRIDE X.805 Trike
MITRE CAPEC
CWE
![Page 19: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/19.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Back to development
• Continuous-prevention developmentBonus: Assert others by looking for defect’s
fix
![Page 20: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/20.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Multiple Tool Evaluation Criteria
• Basic criteria• FN vs. FP• Non-exploitables?
• TP’s vs. testing ground– OWASP SiteGenerator– Stanford SecuriBench
![Page 21: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/21.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Single Tool Evaluation Criteria
• Advanced criteria• NIST SAMATE Evaluation Criteria
– Bug categories (CWE, OWASP, WASC, PCI)– Levels of defense
• 100 * TP / TP + FP + FN (Brian Chess)
![Page 22: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/22.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
The Future
• Hybrid tools and hybrid people?• Logical vs. semantic (Curphey’s flaws vs.
bugs)
![Page 23: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/23.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
RefsManager-type advice: http://codesecurely.org/archive/2007/07/14/the-art-of-managing-
up-when-sucking-up-isn-t-gonna-cut-it.aspxOWASP DC on RIA: http://www.owasp.org/index.php/RIA_Security_SmackdownISM3: http://www.ism3.com SOTA MM’s: http://securitybuddha.com/2007/08/30/software-
security-assurance-state-of-the-art-report/Continuous Integration book - http://www.testearly.comSecurity Metrics: Modelers vs. measurers -
http://safari5.bvdep.com/9780321349989/ch02lev1sec2?imagepage=13ISECOM’s OSSTMM: http://www.isecom.orgMark Curphey – Types of testing: http://securitybuddha.com/2007/09/03/the-art-of-
scoping-application-security-reviews-part-2-the-types-of-testing-2/Promoting Warnings to Errors:
http://safari5.bvdep.com/9780596510237/enabling_useful_warnings_disabling_useless_ones_and_promoti
PMD: http://pmd.sf.net CheckStyle: http://checkstylesf.net FindBugs: http://findbugs.sf.netCT-Eclipse: http://ct-eclipse.tigris.org EMMA: http://emma.sf.net http://www.eclemma.orgBuildix: http://buildix.thoughtworks.com Java metrics: http://metrics.sf.net
![Page 24: By Andre Gironda - OWASP · • OSSTMM – Open-Source Security Testing Methodology Manual – For pen-testers – OSSTMM v3 – Book: Annotated OSSTMM • You have to wait until](https://reader033.fdocuments.net/reader033/viewer/2022051806/5ffce161d4144064df006ab3/html5/thumbnails/24.jpg)
September 5 2007 Automated Thrash Testing Andre Gironda
Refs (cont’d)SecureCoding Mailing-list: http://www.securecoding.org/list/Atlassian (formerly Cenqua) Crucible: http://www.atlassian.com/software/crucible/Concolic testing: http://osl.cs.uiuc.edu/~ksen/cute/Fuzzing in the corporate world, Gadi Evron:
http://events.ccc.de/congress/2006/Fahrplan/events/1758.en.htmlProxy Fuzzing: http://www.darknet.org.uk/2007/06/proxyfuzz-mitm-network-fuzzer-in-
python/GPath with XmlParser and NekoHTML:
http://sylvanvonstuppe.blogspot.com/2007/08/ive-said-it-before-but.htmlCanoo WebTest: http://webtest.canoo.com Jameleon: http://jameleon.sf.netTwill: http://twill.idyll.org MaxQ: http://maxq.tigris.orgOpenQA Selenium, Watir: http://openqa.org TestGen4Web:
http://developer.spikesource.com/wiki/index.php/Projects:TestGen4WebWebDriver: http://code.google.com/p/webdriver/ Apodora: http://www.apodora.orgFalse-positives vs. Non-exploitables: http://sylvanvonstuppe.blogspot.com/2007/04/false-
positives-vs-non-exploitables.htmlBrian Chess & Katrina Tsipenyuk:
http://securitymetrics.org/content/attach/Welcome_blogentry_010806_1/software_chess.ppt