Testing Heuristic Detections
-
Upload
frisksoftware -
Category
Economy & Finance
-
view
717 -
download
1
description
Transcript of Testing Heuristic Detections
![Page 2: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/2.jpg)
What do you need?
The appropriateness of the methodology (or it’s correct application) Repeatability Independently verifiable Validated sample sets
Adherence to safe and ethical practices in handling and testing samples
Understanding of what heuristic detection is (and what it’s not)
![Page 3: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/3.jpg)
A quick word on FP testing
• No ‘tricks’!– Appropriate “ItW” false positive set– Evaluation of FP’s– ‘Grey’/unusual or very strange unlikely files will
tend to penalize heuristic based products
• Defaults• Best settings
![Page 4: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/4.jpg)
Junk / Corrupt files
• Poor sample sets simply reinforce the cycle - the more junk added, the more detected
• Using AV products to determine maliciousness is silly, it simply reinforces the cycle (Kaminski - Eicar 2006?)
![Page 5: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/5.jpg)
“Time to Update”
Product Actual Time to Update / % missed (20 Samples)
Average TtU
X1 1 hour at 100% (20 upd) 1 hour
X2 4 hours at 5% (1 upd) 4 hours
X3 8 hours at %50 (10 upd) 4 hours
X4 30 hours at %20 (5 upd) 6 hours
![Page 6: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/6.jpg)
Actual TtU
Product Actual Time to Update / % missed
Average TtU
(zero removed)
X1 1 hour at 100% 1 hour
X2 4 hours at 5% 4 hours
X3 8 hours at 50% 8 hours
X4 30 hours at %20 30 hours
![Page 7: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/7.jpg)
Mean time
Statistical problems in mean comparison
0
500
1000
1500
2000
2500
3000
0% 10% 20% 30% 40% 50% 60% 70%
No of proactive detections in wk
No of updates in week
Each Dot represents a different product
![Page 8: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/8.jpg)
Lies, Damned Lies and Statistics
• Statistical intgrity is biased, means of more succesful product are calculated over less samples (necessarily). This is not good for comparisons.
• Concentrating on speed of update is surely sending the wrong message to the consumers, giving them the false impression that buying a product that releases a lot of updates very quickly is going to protect them better.
![Page 9: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/9.jpg)
Retrospective (Frozen Update)
• Selection of time period– 6 months?– 3 months?– 1 day?– 1 hour?
• Verification (is it possible to do real time?)
![Page 10: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/10.jpg)
Frozen Update Pt II
• What samples are important?
• Is this a recursive process?– Single snapshot is not necessarily the most
useful information– Performance over time– Sound statistical model
![Page 11: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/11.jpg)
To quote Dr Alan Solomon.
• 1. If something is superb at detecting viruses, it's no use if it gives a lot of false alarms.
• 2. Anything that relies on the user to make a correct decision, on matters that he is not likely to be able to decide about, is useless.
• 3. You can receive something that is *exactly* what the salesman promised to deliver, and it's nevertheless useless.
![Page 12: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/12.jpg)
![Page 13: Testing Heuristic Detections](https://reader038.fdocuments.net/reader038/viewer/2022110308/5575c43ad8b42a312a8b4c20/html5/thumbnails/13.jpg)
Shameless plug
AVIEN Guide to Managing Malware in the Enterprise
http://www.smallblue-greenworld.co.uk/pages/avienguide.html