Filtering Mail with Mail::Audit and Mail::SpamAssassin

Creede Lambardpenguinsinthenight.com

20 August 2002

General Outline:

General Outline:

● How UNIX handles mail

General Outline:

● How UNIX handles mail● A simple understated diatribe against unsolicited

commercial email

General Outline:

● How UNIX handles mail● A simple understated diatribe against unsolicited

commercial email● Why mail filtering is a Good Thingtm

General Outline:

● How UNIX handles mail● A simple understated diatribe against unsolicited

commercial email● Why mail filtering is a Good Thingtm

● If you use Windows . . .

General Outline:

● How UNIX handles mail● A simple understated diatribe against unsolicited

commercial email● Why mail filtering is a Good Thingtm

● If you use Windows . . . ● Using Mail::Audit

General Outline:

● How UNIX handles mail● A simple understated diatribe against unsolicited

commercial email● Why mail filtering is a Good Thingtm

● If you use Windows . . . ● Using Mail::Audit● Using Mail::SpamAssassin

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

How Unix handles your mail

.forward to another mail address:

me@myotherisp.com

How Unix handles your mail

Piping to another program:

| vacation

Does this look familiar?

spam

spam

● Unsolicited commercial email

spam

● Unsolicited commercial email– Sent in bulk

spam

● Unsolicited commercial email– Sent in bulk– Directly or indirectly advertises a product or service

spam

● Unsolicited commercial email– Sent in bulk– Directly or indirectly advertises a product or service– Not requested by recipient

spam

● Unsolicited commercial email– Sent in bulk– Directly or indirectly advertises a product or service– Not requested by recipient

● Not necessarily mail you don't want . . .

spam

● Unsolicited commercial email– Sent in bulk– Directly or indirectly advertises a product or service– Not requested by recipient

● Not necessarily mail you don't want . . .– Although for purposes of this presentation we'll treat them

the same.

When Spamtm is acceptable

When Spamtm is acceptable

spam is a Bad Thingtm

spam is a Bad Thingtm

● It shifts the burden of costs to the recipient

spam is a Bad Thingtm

● It shifts the burden of costs to the recipient● It clogs the Net

spam is a Bad Thingtm

● It shifts the burden of costs to the recipient● It clogs the Net● It wastes your time

spam is a Bad Thingtm

● It shifts the burden of costs to the recipient● It clogs the Net● It wastes your time● Items/services advertised through spamming tend to

be of questionable value

spam is a Bad Thingtm

● It shifts the burden of costs to the recipient● It clogs the Net● It wastes your time● Items/services advertised through spamming tend to

be of questionable value● The vast majority of it is fraudulent

Dealing with spam

Dealing with spam

● Ignore it

Dealing with spam

● Ignore it . . . and hope it goes away

Dealing with spam

Dealing with spam

● Ignore it . . . not an option

Dealing with spam

● Ignore it . . . not an option● Just hit Delete . . .

Dealing with spam

● Ignore it . . . not an option● Just hit Delete . . . The damage is already done

Dealing with spam

● Ignore it . . . not an option● Just hit Delete . . . The damage is already done● Filter it as early as possible in its life cycle

Dealing with spam

● Ignore it . . . not an option● Just hit Delete . . . The damage is already done● Filter it as early as possible in its life cycle● Filter it as it's trying to enter your machine

If you use Windows . . .

Mail filtering

Mail filtering

| /home/you/mailfilter

Mail filtering apart from spam filtering

Mail filtering apart from spam filtering

● Separating mailing lists into their own folders

Mail filtering apart from spam filtering

● Separating mailing lists into their own folders● News-to-mail gateways

procmail

procmail

● Advantages:

procmail

● Advantages:– Well-established

procmail

● Advantages:– Well-established– Lots of sample scripts

procmail

● Advantages:– Well-established– Lots of sample scripts

● Disadvantages:

procmail

● Advantages:– Well-established– Lots of sample scripts

● Disadvantages:– Arcane syntax

procmail

● Advantages:– Well-established– Lots of sample scripts

● Disadvantages:– Arcane syntax– Like learning a new language . . .

procmail

● Advantages:– Well-established– Lots of sample scripts

● Disadvantages:– Arcane syntax– Like learning a new language . . .– And it's not Perl!

Mail::Audit

Mail::Audit

● Written by Simon Cozens

Mail::Audit

● Written by Simon Cozens

procmail is nasty. It has a tortuous and complicated recipe format, and I don't like it. I wanted something flexible whereby I could filter my mail using Perl tests.

- Simon Cozens, from the Mail::Audit perldoc

Mail::Audit

● Written by Simon Cozens● Based on audit_mail and deliverlib by Tom

Christiansen

Mail::Audit

● Written by Simon Cozens● Based on audit_mail and deliverlib by Tom

Christiansen● It's Perl!!!!!!!!!!!!!!!

Mail::Audit

● Written by Simon Cozens● Based on audit_mail and deliverlib by Tom

Christiansen● It's Perl!!!!!!!!!!!!!!!● A module, not a standalone program

How Mail::Audit Works

Parsing mail

Parsing mail

● Mail::Internet object

Parsing mail

● Mail::Internet object● Parse by:

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines– Subject

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines– Subject– Absence, presence or content of headers

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines– Subject– Absence, presence or content of headers– Body text

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines– Subject– Absence, presence or content of headers– Body text

● Anything can be parsed

Parsing mail

● Mail::Internet object● Parse by:

– From, To or CC lines– Subject– Absence, presence or content of headers– Body text

● Anything can be parsed– Using Mail::Internet::as_string

Installation

Installation

● Download and install Mail::Audit from CPAN

Installation

# perl -MCPAN -e shell

cpan> install Mail::Audit

Installation

● Download and install Mail::Audit from CPAN● Create .forward file

Installation

| /home/creede/mailfilter

Installation

● Download and install Mail::Audit from CPAN● Create .forward file● Create filter file

Installation

#!/usr/bin/perl

use Mail::Audit;

my $mail = new Mail::Audit;

Installation

#!/usr/bin/perl

use Mail::Audit;

my $mail = new Mail::Audit;

my $from = $mail->from;

my $to = $mail->to;

my $cc = $mail->cc;

my $subject = $mail->subject;

Installation

#!/usr/bin/perl

use Mail::Audit;

my $mail = new Mail::Audit;

my $from = $mail->from;

my $to = $mail->to;

my $cc = $mail->cc;

my $subject = $mail->subject;

my $_body = $mail->body;

my $body = join(“\n”, @$body);

Installation

#!/usr/bin/perl

use Mail::Audit;

my $mail = new Mail::Audit;

my $from = $mail->from;

my $to = $mail->to;

my $cc = $mail->cc;

my $subject = $mail->subject;

my $_body = $mail->body;

my $body = join(“\n”, @$body);

my $xloop = $mail->get('X-Loop');

Installation

#!/usr/bin/perl

use Mail::Audit;

my $mail = new Mail::Audit;

my $from = $mail->from;

my $to = $mail->to;

my $cc = $mail->cc;

my $subject = $mail->subject;

my $_body = $mail->body;

my $body = join(“\n”, @$body);

my $xloop = $mail->get('X-Loop');

my $message = $mail->{obj}->as_string;

Installation

● Download and install Mail::Audit from CPAN● Create .forward file● Create filter file● Remember to chmod 0755!

Mail disposition

● $mail->accept– Accepts mail into default inbox

Mail disposition (continued)

if ($mail->from =~ /mom@applepie.com/) {

$mail->accept;

}

Mail disposition (continued)

● $mail->accept(“/path/to/alternate/mailbox”)– Accepts mail into a non-default mailbox

Mail disposition (continued)

my $maildir = “/home/me/mail”;

if ($mail->subject =~ /spug/i) {

$mail->accept(“$maildir/spug-list”);

}

Mail disposition (continued)

● $mail->pipe(“/path/to/external/program”)– Pipes mail through the specified program

Mail disposition (continued)

if ($mail->subject =~ /keplerian/i) {

$mail->pipe(“/home/creede/parse_kepler”);

}

Mail disposition (continued)

● $mail->resend(“someguy\@otherisp.com”)– Sends the mail in its entirety to another address

Mail disposition (continued)

if (is_419($message)) {

$mail->{noexit} = 1;

$mail->put_header('X-Loop',

'creede@penguinsinthenight.com');

$mail->put_header('To', "$to (forwarded --

no monetary loss -- for your files)");

$mail->resend("uce\@ftc.gov");

$mail->resend("419.fcd\@usss.treas.gov");

$mail->{noexit} = 0;

$mail->ignore;

}

Mail disposition (continued)

● $mail->reject($reason)– Rejects the mail, returning it to the sender with the

(optional) reason specified

Mail disposition (continued)

if (is_murky($mail)) {

$mail->put_header('X-Loop',

'creede@penguinsinthenight.com');

$mail->reject("I don't like spam.");

}

Mail disposition (continued)

● $mail->ignore– Consigns the mail to the bit bucket

Mail disposition (continued)

# kill off Korean spam

if ($body =~ /ks.c/i) {

$mail->ignore;

}

Mail::SpamAssassin

Mail::SpamAssassin

● Header analysis

Mail::SpamAssassin

● Header analysis● Text analysis

Mail::SpamAssassin

● Header analysis● Text analysis● Blacklists

Mail::SpamAssassin

● Header analysis● Text analysis● Blacklists● Vipul's Razor

Mail::SpamAssassin – Installation

● Download and install Mail::SpamAssassin from CPAN

Mail::SpamAssassin – Installation

# perl -MCPAN -e shell

cpan> install Mail::SpamAssassin

Mail::SpamAssassin – Installation

#!/usr/bin/perl

use Mail::Audit;

use Mail::SpamAssassin;

my $mail = new Mail::Audit;

my $spamtest = new Mail::SpamAssassin;

my $status = $spamtest->check($mail);

if ($status->is_spam()) {

$mail>accept(“/home/you/spamtrap”);

}

Mail::SpamAssassin – Configuration

● Load configuration from /etc/mail/spamassasin.conf or /home/you/.spamassassin/user_prefs

Mail::SpamAssassin – Configuration

# SpamAssassin user preference file

#

required_hits 4

#

# default is 5

#

whitelist_from mom@applepie.com

blacklist_from scuzzball@spamspewer.com

score USER_AGENT_AOL 1.00

Paul Graham's Plan for Spam

Paul Graham's Plan for Spam

madam 0.99

promotion 0.99

republic 0.99

republic 0.99

shortest 0.047225013

mandatory 0.047225013

standardization 0.07347802

2600 0.0813768

sorry 0.08221981

supported 0.09019077

URLs for more information

URLs for more information

● Internet Mail

http://www.imc.org/rfcs.html

URLs for more information

● Internet Mail

http://www.imc.org/rfcs.html● Mail::Audit

http://simon-cozens.org/writings/mail-audit.html

URLs for more information

● Internet Mail

http://www.imc.org/rfcs.html● Mail::Audit

http://simon-cozens.org/writings/mail-audit.html● Mail::SpamAssassin

http://www.spamassassin.org/

http://www.deersoft.com (Outlook)

URLs for more information

● Internet Mail

http://www.imc.org/rfcs.html● Mail::Audit

http://simon-cozens.org/writings/mail-audit.html● Mail::SpamAssassin

http://www.spamassassin.org/

http://www.deersoft.com (Outlook)● Paul Graham's Plan for Spam

http://www.paulgraham.com/spam.html

URLs for more information● Internet Mail

http://www.imc.org/rfcs.html● Mail::Audit

http://simon-cozens.org/writings/mail-audit.html● Mail::SpamAssassin

http://www.spamassassin.org/

http://www.deersoft.com (Outlook)● Paul Graham's Plan for Spam

http://www.paulgraham.com/spam.html● And of course Google.com!

Questions?

Thank you!

creede@penguinsinthenight.com

http://www.penguinsinthenight.com/spamtalk