Using Clustering to Detect and Mitigate Spam Distribution · 2012. 10. 5. · An anti-spam...

13
Using Clustering to Detect and Mitigate Spam Distribution Andrey Bakhmutov Kaspersky Lab [email protected]

Transcript of Using Clustering to Detect and Mitigate Spam Distribution · 2012. 10. 5. · An anti-spam...

  • Using Clustering to Detect and Mitigate Spam Distribution

    Andrey Bakhmutov

    Kaspersky Lab [email protected]

  • Email clustering system requirements

    An anti-spam clustering system should:

    • be large-scale and distributed

    • use an efficient, secure data model

    - small data size

    - inability to restore original content

    • adopt an efficient algorithm for real-time clustering at high speed

    • employ extra tools to identify spam clusters

  • Email processing stages

    text/

    html

    applic/

    msword

    text/

    plain

    normalized

    text

    0x13BC7091

    0x0012A7E1

    0x901A78FC

    0x6380901A

    0xB278E002

    0xF2898321

    0x440D1105

    0x06DD2129

    Email (mime) Shingles

    text/

    plain

    normalized

    text

    chunks

    chunks

  • Shingles

    find out why hcg works

    extremely effective rapid

    weight loss supplement

    and where you can order

    hcg online from verified

    and trusted online http

    hcgkiloshot

    0xC05F5857

    0x2EA6814D

    0xB60257F0

    0xA18E4FCD

    0x20167ADD

    0x31E713B8

  • Clustering database

  • Grouping capabilities

  • System architecture

    UDS 1

    Gateway

    Spam traps

    UDS 2 Email

    sources Clients

  • Spam vs. mailing list sources

    ip-109-38.hyper.net.id

    ABTS-mum-dynamic-251.90.170.122.airtelbroadband.in

    Static-177.143.195.14.tataidc.co.in

    95x153x190x63.kubangsm.ru

    112-79-36-62.live.vodafone.in

    smtp293.usndr.com

    smtp289.usndr.com

    smtp131.usndr.com

    smtp91.usndr.com

    smtp171.usndr.com

  • Messages with auto-signatures

    Here's to an all Manchester Europa League Final!!!!!

    CONFIDENTIALITY NOTICE: This e-mail message, including

    any attachments, is for the sole use of the intended

    recipient(s), and may contain confidential and privileged

    information. Any unauthorized review, use, disclosure or

    distribution is prohibited. If you are not the intended

    recipient, please contact the sender by reply e-mail and

    destroy all copies of the original message.

    Chelsea mailing list

    [email protected]

    http://jesternix.net/mailman/listinfo/chelsea_jesternix.net

  • Message refinement

    Here's to an all Manchester Europa League Final!!!!!

    CONFIDENTIALITY NOTICE: This e-mail message, including

    any attachments, is for the sole use of the intended

    recipient(s), and may contain confidential and privileged

    information. Any unauthorized review, use, disclosure or

    distribution is prohibited. If you are not the intended

    recipient, please contact the sender by reply e-mail and

    destroy all copies of the original message.

    Chelsea mailing list

    [email protected]

    http://jesternix.net/mailman/listinfo/chelsea_jesternix.net

  • Quarantine usage

    quarantine

    junk

    mailbox

    email

    UDS 2

    UDS 1

    clusters

  • Spam campaign mitigation

    objects

    seconds 6 20 31

    15

    25

    84 UDS2

    UDS1

    missed

    quarantined rejected

  • THANK YOU

    Andrey Bakhmutov

    Kaspersky Lab [email protected]


A System for High-Throughput Spam Analysis and …rpelizzi/spam_slides.pdfObtain statistics about redirection ... Riccardo Pelizzi A System for High-Throughput Spam Analysis and Clustering.

A System for High-Throughput Spam Analysis and …rpelizzi/spam_slides.pdfObtain statistics about redirection ... Riccardo Pelizzi A System for High-Throughput Spam Analysis and Clustering.

International Telecommunication Union...International Telecommunication Union 4 Spam spam spam zWireless spam: SMS, MMS, email over mobile devices…(and SMS scams) z“Spim”: spam

International Telecommunication Union...International Telecommunication Union 4 Spam spam spam zWireless spam: SMS, MMS, email over mobile devices…(and SMS scams) z“Spim”: spam

ΑΜ.: 1130 µΒίκινγκς τραγουδούσε “Spam, spam, spam, …”, παρενοχλώντας οποιαδήποτε συνοµιλία µεταξύ των υπολοίπων

ΑΜ.: 1130 µΒίκινγκς τραγουδούσε “Spam, spam, spam, …”, παρενοχλώντας οποιαδήποτε συνοµιλία µεταξύ των υπολοίπων

Spam and Anti-Spam By Aditi Desai Yousuf Haider. Agenda Introduction Purpose of Spam Types of Spam Spam Techniques Anti spam Why Spam is so Easy Anti.

Spam and Anti-Spam By Aditi Desai Yousuf Haider. Agenda Introduction Purpose of Spam Types of Spam Spam Techniques Anti spam Why Spam is so Easy Anti.

Spam Spam Spam Spam

Spam Spam Spam Spam

Affinity Clustering: Hierarchical Clustering at Scalepapers.nips.cc/paper/7262-affinity-clustering-hierarchical-clustering-at-scale.pdf · Affinity Clustering: Hierarchical Clustering

Affinity Clustering: Hierarchical Clustering at Scalepapers.nips.cc/paper/7262-affinity-clustering-hierarchical-clustering-at-scale.pdf · Affinity Clustering: Hierarchical Clustering

Vorbesprechung Mathe IIIvera/Vorbesprechung.pdf · Klassi kation Textkategorisierung Spam Filter Stimmungsanalyse (Sentiment Analysis) Clustering Welche Texte / W orter sind ahnlich

Vorbesprechung Mathe IIIvera/Vorbesprechung.pdf · Klassi kation Textkategorisierung Spam Filter Stimmungsanalyse (Sentiment Analysis) Clustering Welche Texte / W orter sind ahnlich

Spam, Spam, Spam, Spam….

Spam, Spam, Spam, Spam….

Detecting Spam and Spam Responses

Detecting Spam and Spam Responses

Training Employees to Recognise & Avoid Advanced Threats · Recognise and Mitigate Advanced Threats STAR-302 General Interest . ... return to battle the daily spam stream. Engaging

Training Employees to Recognise & Avoid Advanced Threats · Recognise and Mitigate Advanced Threats STAR-302 General Interest . ... return to battle the daily spam stream. Engaging

NEW ABUSE REPORT - AFNIC · Google Safe Browsing Domain name Abuse spam phishing spam phishing spam Unwanted s spam spam malware GURID Registrar name Potential abuses Creation date

NEW ABUSE REPORT - AFNIC · Google Safe Browsing Domain name Abuse spam phishing spam phishing spam Unwanted s spam spam malware GURID Registrar name Potential abuses Creation date

%% · Highdim. data Locality# sensive# hashing# Clustering# Dimensional ity# reducon# Graph$$ data PageRank,# SimRank# Community# DetecOon# Spam# DetecOon# Infinite

%% · Highdim. data Locality# sensive# hashing# Clustering# Dimensional ity# reducon# Graph$$ data PageRank,# SimRank# Community# DetecOon# Spam# DetecOon# Infinite

Thwarting Emotet email conversation thread …...•Scikit-learn Python machine learning library Clustering emails • Qian et al. (2010). A Case for Unsupervised-learning-based Spam

Thwarting Emotet email conversation thread …...•Scikit-learn Python machine learning library Clustering emails • Qian et al. (2010). A Case for Unsupervised-learning-based Spam

Hexamail guard anti-spam server spam filtering software - index

Hexamail guard anti-spam server spam filtering software - index

SPAM And SpamAssassin - haifux.orghaifux.org/hebrew/lectures/155/spam.pdf · The SPAM arms race SPAM filtering is a unique problem in AI. Moving target – SPAM keeps changing. –

SPAM And SpamAssassin - haifux.orghaifux.org/hebrew/lectures/155/spam.pdf · The SPAM arms race SPAM filtering is a unique problem in AI. Moving target – SPAM keeps changing. –

CASL vs CAN-SPAM - Canada’s Anti‐Spam Law

CASL vs CAN-SPAM - Canada’s Anti‐Spam Law

Clustering Clustering

Clustering Clustering

SlidesbyJeffreyD.Ullman · Highdim. data Locality) sensive) hashing) Clustering) Dimensional ity) reducon) Graph’’ data PageRank,) SimRank) Community) Detec-on) Spam) Detec-on)

SlidesbyJeffreyD.Ullman · Highdim. data Locality) sensive) hashing) Clustering) Dimensional ity) reducon) Graph’’ data PageRank,) SimRank) Community) Detec-on) Spam) Detec-on)

EK Ch 17: Power laws and rich-get-richer phenomena (with an application of Web Spam detection Spam, Damn Spam and Statistics ) Spam, Damn Spam and Statistics.

EK Ch 17: Power laws and rich-get-richer phenomena (with an application of Web Spam detection Spam, Damn Spam and Statistics ) Spam, Damn Spam and Statistics.

Claudiu MUSAT, Ionut GRIGORESCU, Carmen MITRICA, Alexandru TRIFAN Spam Clustering using Wave Oriented K Means.

Claudiu MUSAT, Ionut GRIGORESCU, Carmen MITRICA, Alexandru TRIFAN Spam Clustering using Wave Oriented K Means.