1

Winning the War on SpamNorthwest Academic Computing ConsortiumOnline Content: Policy, Strategy and Support

June 5-6, 2003, Portland, Oregon

Joe St Sauver, Ph.D. (joe@oregon.uoregon.edu)Director, User Services and

Network ApplicationsUniversity of Oregon Computing Center

http://darkwing.uoregon.edu/~joe/spamwar/

2

I. Introduction

3

Our Objective:Email "the Way It Used to Be"• Our objective: We want email to be "the

way it used to be." That implies that spamwill be virtually non-existent, and thatnormal academic communication can takeplace with minimal games -- no need toconceal your address from most of theworld, no worries about posting to mailinglists w/o munging your address, nochallenge-response BS, etc….

4

Calibrate Your Expectations• The war against spam won't be won

overnight. It took many years for spam toget really bad; it will similarly take sometime to get spam cleaned up. However,having said that, we can easily knock spamdown to 1%, or 1/10th of 1% of what itwould be if it were left unfiltered byfollowing the approach outlined later in thistalk. Yes, that would not be "perfect," but itmay be "good enough" (at least for now).

5

What We'll Cover Today• -- Just how bad is the spam problem?

-- The decision to block (or not block) spam and the potential costs of that decision-- Deciding between content-based and non-content-based filtering-- Filtering traffic from spam-tolerant providers and from insecure hosts, and-- Training your users to report spam properly

6

Sticking To The Script• Because we have a lot to cover, and because

many spam fighting folks from yourinstitutions who may not be attending today,I've prepared this talk in some detail andwill try to "stick to the script."

• This is a good news/bad news thing: ifyou're looking at this presentation after thefact, you'll be able to follow what wascovered; the bad news is that if you're in theaudience today, there won't be a lot of"surprises" in the talk not in this handout.

7

II. How Bad Is The SpamProblem? Is It Really Time To

Call It A "War"?Some may be reluctant to admit

they're under siege until the enemybegins flinging rotted cows over thewall with a trebuchet, but for the restof us the signs are quite clear that yes,spammers are now waging war on us.

8

It's Hard To Get GoodNumbers For Spam Volume...

• Something I realized while listening to partsof the recent three day FCC Spam Forum(http://www.ftc.gov/bcp/workshops/spam/)is that while spam is ubiquitous, it is hard toget good numbers concerning the totalvolume of spam that users see.

There are many reasons for this...

9

Reasons Why Spam Volume IsGenerally Poorly Quantified

• -- The percentage of email that's spam willvary from user to user, from site to site, andfrom day to day (so please beware ofrelying on statistics from one user'sexperience or even one site's experience)

-- Not all sites even have the ability todistinguish spam from non-spam messages

10

Reasons Why Spam Volume IsGenerally Poorly Quantified (2)• -- Sites that can identify incoming email as

spam will typically block email from thatsource (either in real time or post hoc).

That blocking process will typically reducethe amount of spam seen from that sourceby that site (e.g., one attempted-but-blockedemail connection may represent 1,000 or10,000 or more avoided pieces of spam).

11

Reasons Why Spam Volume IsGenerally Poorly Quantified (3)• -- Some spam will almost always "slip

through filters;" likewise some legitimateemail will almost always be falsely taggedby filters as spam (this is what's usuallyreferred to as a "false positive"). These sortof classification errors affect estimatedspam volume (upwards or downwards).

• Spam volume is constantly increasing, soany spam volume estimate made today willbe low tomorrow.

12

One Industry Estimate• Nonetheless, one leading anti-spam firm

which is able to categorize and measurespam from a large number of clients (beforefiltering that email for them) estimates thatas of June 2003, 77.4% of all email is nowspam. See: http://www.postini.com/stats/

That's three, nearly four, pieces of spam foreach legitimate piece of email. That's a lotof spam.

13

Gartner Sez...• "There is now 16 times as much spam on the

Internet as there was just two years ago"[http://www.ecommercetimes.com/perl/story/16874.html (March 26, 2002)]This implies a 6 month doubling time, e.g.:Time=0 volume=XTime=6 months volume=2XTime=12 months volume=4XTime=18 months volume=8XTime=24 months volume=16X[Time="now" volume=>64X?]

14

Other Spam Growth Estimates• Brightmail.com tracks what they refer to

as "spam attacks." Their graph looks like:

15

But How Messages AreGetting Sent?

• We can talk about the percentage of mailthat's spam, or spam volume doublingtimes, but how many actual pieces of mailget sent on a typical day?

• If I were to ask you to guess how muchemail Yahoo sends, or Hotmail, or AOL,what would you say? How many emailssome of the higher volume bulk emailerssend? Could you give me some numbers?

16

Some Top Email Senders'Message Volume Per Day

• http://www.senderbase.com/ estimates:-- Yahoo: 841 million/day!-- Hotmail: 532 million/day-- AOL: 331 million/dayFor comparison, some entities you'veprobably never heard of have traffic running:-- "shoppersville.net:" 288.6 million/day-- "ew01.com" sends 191.2 million/day -- "nnt6us.com" sends 188.9 million/day

• Those volumes explain a number of things...

17

Immense Mail Volumes ==>Limited Filtering Options

• The immense volume of real mail we'retalking about on a daily basis means that forthe largest of ISPs there are limited(outbound and inbound) spam filtering andabuse response options that scale to handlethe volume of email they're dealing with.

• 841 million messages/day = nearly 10,000messages a second, around the clock(except that load actually has peaks andtroughs during the day)

18

Immense Mail Volumes ==> SomeISPs are Making Lots of Money

By Hosting Spammers• The immense volumes of bulk mail that are

being sent also imply that some carriers areknowingly "bulk email friendly." There issimply no way that these ISPs could fail toknow that they have customers sendinghuge volumes of email. ISPs knowinglyhost bulk email customers because thoseguys represent very lucrative accounts.

19

Spam And Exploitation OfVulnerable Systems

• Of course, there are some spammers whoare so odious that even cash-hungry carrierswon't sell them service. Those guys juststeal what they need, with most of themexploiting insecure systems to transmit theirspam.

• For example, I'm now tracking over 300thousand open/abusable proxy servers,many of which are a general security risk(as well as serving as a conduit for spam).

20

Why Don't Those VulnerableHosts Get Fixed?

• Imagine that you are in charge of a largeISP's abuse desk. (You poor person!) Everymorning when you come to work, there arethousands of new complaints aboutcustomers with problems -- insecure hoststhat have been compromised by hackers,virus infested systems, open relays, openproxies, you name it. No matter how hardyou work, more keep coming, day after day.You try to prioritize but you never catch up.

21

Why Don't Those VulnerableHosts Get Fixed? (2)

• Moreover, management tells you that youcan't simply turn those users off -- these arepaying customers we're talking about afterall!! (and how much revenue comes in fromthose folks who are complaining aboutgetting spammed, hmm?)

• As spam overwhelms many ISP abusedesks, a culture of ignoring all securityproblems arises; spam and other securityproblems seem to track very well.

22

So Yes, We Are At War...• Spammers are hammering your mail

servers hard and will not voluntarily stop.• The problem is becoming increasingly

serious (e.g., the trend line is definitelyupward)

• Spam may only be the most visiblesymptom of many distributed securitythreats you are already constantly facing.

• A reasonable person would probably takesteps to protect their users and systems.

23

III. Deciding To Act OnThe Spam Problem

24

TANSTAAFL• The conclusion that you should take action

against spam may be pretty much a matterof common sense at this point, but thedecision to do so won't be without pain.(There Ain't No Such Thing As A FreeLunch).

Understanding the tangible and intangiblecosts associated with the decision to fightspam will be important.

25

The "Collateral Damage" or"False Positive" Problem

• The most fundamental cost of blockingspam is the potential for misclassificationand rejection of real, non-spam messages byanti-spam measures.

• This is normally called "collateral damage"or the "false positive" problem, and is oneof the true (and unavoidable) costs ofblocking spam.

26

When classifying mail, 4 thingscan happen (2 of which are bad):

• Actual Case You Believed The Message Was-- Spam Not Spam [oops! spam got by]

Spam [correct classification]

-- Not Spam Not Spam [correct classification]Spam [oops! false positive]

• We can get fewer false positives if we're morewilling to let more spam slip through, OR lessspam if we can accept more false positives. Wecan't minimize both objectives simultaneously.

27

Quantifiable Costs• Besides the pain associated with

misclassifying real email (and/or failing tofilter some spam messages), fighting spamwill also consume direct personnel andcapital resources.

• It may also come with some indirect (butpotentially substantial) costs.

28

Costs Of Fighting Spam:People's Time

• One real cost of fighting spam is the cost ofpersonnel, including:-- management time working out policies, handling complaints/inquiries, etc.-- systems staff time (configuring filters, etc.)-- end user time spent reporting spam-- user support/postmaster/abuse desk time

• Because nobody hires dedicated anti-spampersonnel, these personnel costs are largelyignored as existing staff "just take care of it."

29

Costs Of Fighting Spam:Hardware

• Fighting spam may entail real hardware costs:-- you may need to accelerate hardware upgrade schedules to insure that you have the CPU and disk space that some filtering approaches may require-- on the other hand, depending on how you filter, you may see a decrease in the rate at which you need to do system upgrades, as the amount of spam which gets delivered and stored goes down.

30

Costs Of Fighting Spam:Software/Services

• When planning the cost of fighting spam, besure to also figure in the cost of anti-spamsoftware products, black list subscriptions,and other software and services.

• What do all these spam-fighting costs totalup to? Lacking better data, assume you'llspend the same amount fighting spam thatyou spend on (antivirus products pluspersonal firewall software) (but this is JUSTa wild rule of thumb). YMMV.

31

Costs Of Fighting Spam:Indirect Costs

• Blocking some email may also entail someindirect institutional costs which may(ironically) dwarf direct out of pocket spamfighting costs.

32

Some Potential CustomersMay End Up Getting Blocked

• For example, if we block some "spam"directed at our admissions office, mightour admissions folks miss requests forinformation from potential enrollees?What's the net cost to the institution if welose tuition revenue from ten (or a hundred)potential out of state students because we'reblocking their inquiry email? [EstimatedUO non-resident full time tuition and fees,2003-2004, run $16,416 per academic year.]

33

Blocking Spam == Censorship?• While trying to block spam in good faith,

you may be accused of censorship orinterfering with academic freedom.

• Some approaches that may diffuse this sortof potentially explosive issue include:-- writing your AUP carefully to cover this-- allowing individual users to "opt out" from all filtering if they desire to do so-- delivering all email, but delivering what's believed to be spam to a different folder than presumptive non-spam email

34

Liability Issues?• Are there liability issues if we don't deliver

all email?• Technical users of email used to understand

that email delivery was NOT assured, andthat sometimes email would NOT getthrough. If it did, great, if it didn't, you'dpick up the phone... Now-a-days, though,many email users seem to assume that emailis an assured delivery service (even thoughit isn't) because it will usually get through…

35

Be Particularly Careful WithCampus M.D.'s, Lawyers, etc.

• Under the Federal ECF(https://ecf.dcd.uscourts.gov/ ) email maynow be used to transmit notices of legalpleadings. If email of that sort is sent to aUniversity attorney and fails to get through,a default judgement may get entered whenhe misses a scheduled hearing.

• Or consider the patient of a teachinghospital surgeon who is unable to email herdoc about her "chest pains," and then dies.

36

What If We Just Do Nothing?• Doing nothing is equally fraught with

potential problems…-- Spam has the potential to act as a denial of service attack against "real" mail. A) Real messages may easily get missed in amongst all the spam. B) Accounts may go over quota from spam, and begin bouncing all email. C) Spam sent to mailing lists you host may get sent on to subscribers, who may then unsubscribe from those lists.

37

Doing Nothing (2)• -- If you choose to do nothing about spam,

there will be a tremendous amount of wasted staff time as staff deal with the spam which they've been sent (plus the temptation to waste still more work time on non-work-related "content" being advertised in the spam they receive).

What's a half hour or hour a day per employee worth to your school?

38

Doing Nothing (3)• -- If you do nothing about spam, users will

create email accounts on 3rd party services which offer some sort of filtering. Do you really want institutional business being done from Hotmail? Nah. Others will select and install their own spam filtering solution (good, bad or indifferent) without consulting you or anyone else.

Doing nothing ==> email chaos reigns.

39

Doing Nothing (4)• -- Keeping in mind that much spam may

contain particularly objectionable sexuallyrelated content, allowing spammersunfettered access to your faculty and staffincreases the chance that you may be thesubject of a hostile workplace sexualharassment suit. [see: http://news.com.com/2100-1032-995658.html -- I swear I don'tmake this stuff up!]

40

Doing Nothing (5)• If everyone else EXCEPT you filters, you

are going to see a tremendous amount ofspam as spammers give up on the guys whofilter, and devote their attentions solely tothe guys who are left.

41

So What Should Be Done?• Finesse the problem. :-)• Your best bet is probably going to be to

spam filter ALL accounts by default, butallow some accounts to "opt out" and beexempt from institutionally- performedfiltering on request.

42

Talk To Your Legal and SeniorAdministrative Folks

• One procedural note: whatever you decideto do about spam, be sure to talk to youruniversity's attorneys and your senioradministration folks before you implementany spam filtering strategy. Spam tends tobe highly newsworthy, and there's a distinctchance you'll have a "Chronicle of HigherEducation" moment if things go awry. DoNOT surprise your staff attorneys or yourChancellor/President/Provost.

43

III. Picking Your DefensiveAnti-Spam Strategy

content-based filtering vs. non-content-based filtering

44

Content Based Filtering (CBF)vs. Non-CBF filtering (NCBF)

• As you harden your systems against spam,you have a fundamental decision you needto make early on: am I going to do content-based filtering (CBF), or non-content-basedfiltering (NCBF)?

• Put another way, do I care about what's inthe body of the message, or just about howthe message arrived and possibly what's inthe message headers?

45

One Point In Favor Of CBF...• The biggest point in favor of CBF is that

there is some spam which has relativelyconstant, readily detectable, and triviallyfilterable based on its content.

• If you DON'T do CBF and "easilyidentifiable" spam ends up gettingdelivered, folks will say, "How come a'smart' computer can't ID obvious spammessages when I can easily do so?" This isa (sort of) legitimate complaint.

46

Another Advantage Of CBF• A second advantage of doing content based

filtering is that it allows you to selectivelyaccept some content from a given trafficsource, while rejecting other content fromthat same source.

• This can be useful if you're dealing with alarge provider (such as a mailing list hostingcompany) that has both legitimate andspamy customers, and you don't want to endup dumping the legitimate traffic along withthe spam.

47

CBF Issues: False Positives• On the other hand, one of the biggest issue

with CBF is the problem of false positives(mentioned previously). Because CBF usesa series of rubrics, or "rules of thumb," it ispossible for those rubrics to be falselytriggered by content that "looks like" spamto the filtering rules but which actually isn'tspam. For example, some (relatively crude)content based filters make it impossible fora correspondent to include certain keywordsin a legitimate email message.

48

Using Scoring to MinimizeFalse Positives

• Most content-based-filtering software,however, does "scoring" rather than justusing a single criteria to tag spam. Forexample, a message in ALL CAPS mightgets 0.5 points; if it also mentions millions ofdollars and Nigeria, it might gets another 1.2points; etc. Messages with a total score thatexceeds a specified threshold get tagged asspam; the mere presence of a single badkeyword alone typically wouldn't be enough.

49

CBF Issues: The Arms Race• Because content-based filtering attempts to

exploit anomalous patterns present in thebody of spam messages, there's acontinuous arms race between those lookingfor patterns, and those attempting to avoidfiltering.

• This process of chasing spam patterns andmaintaining odd anti-spam heuristic rulesetsis not particularly elegant, and violates thetraditional desire for parsimonious solutionsto scientific problems.

50

CBF And The Need For"Security Through Obscurity"• For content based filtering to work well, the

filters used may need to be not-well-known.For example, hypothetically, if a spammerknows that using %-encoded URLs willresult in their mail getting rejected, theywon't use %-encoded URLs in their spam.Thus, CBF rule efficacy may be inverselyproportional to the notoriety of those rules,and preserving the effectiveness of rulesmay require keeping the rules used "secret."

51

CBF And Simplicity• Another issue with CBF is the complexity

problem: the reason why a piece of emailgets filtered should be triviallycomprehensible. Because many CBF's useinherently complex rules, explaining thoserules to a user (or to their remotecorrespondent who is being filtered) can bea challenge (assuming you're allowed todisclose the basis for a given message beingrejected). (Some products add X-headerswith an explanation of rules that were hit)

52

CBF Issues: System Load• Because CBF applies <n> unique rules to

the body of each message that's received, asthe number of filtering rules increases, orthe size of the message body increases, orboth, processing tends to slow down.

• Yes, filtering rules can be applied in orderof efficacy, and arbitrary decisions can bemade to limit message body examination tojust the first 100KB/message (or whatever),but the scaling problem remains a real one.

53

CBF And The Need To Use AllRules On Each Legitimate Email• Ironically, the more legitimate email you

have, the worse CBF tends to perform.• To understand this, note that filtering spam

is a logical "OR" process -- potentially onlyone filter condition needs be met for a spamto be junked. On the other hand, acceptinglegitimate email is a logical "AND" process,and requires that a legitimate message betested against (and successfully pass!) ALLpotential filtering tests before acceptance.

54

CBF And Privacy• Doing content based filtering also implicitly

seems "more intrusive" to users than doingnon-CBF.

• Even when CBF is done in a fullyautomated way, users may still be "creepedout" at the thought that their email is being"scanned" for keywords/spam patterns, etc.

• "Big Brother" is a powerful totem, whoseinvocation should be avoided at all costs.

55

Alternatives to CBF• By now you may have the idea that I'm not

a big fan of content-based filtering. You'reright! So what's the alternative?

• The alternative is to focus on systems thatare insecurely configured (and which arethereby unintentionally allowing spam to besent), and providers that are bulk-emailfriendly (thereby intentionally allowingspam to be sent). If you block traffic fromthem, spam levels will drop substantially.

56

IV. It's Not What's InThe Message, It's Where The

Message Comes FromThat Matters

Understanding the efficiencyand elegant simplicity of using

non-content-based spamfiltering strategies

57

Understanding Spammers'Fundamental Problem

• Spammers face a fundamental problemwhen sending spam: every piece of spamthey send announces where it came from.If I know where a message came from (andI always do, at least in terms of the machinethat actually handed me that message), I canput filters in place to block future mail fromthat source. Thus, every time a spammeruses an address to deliver a piece of spam,he puts further use of that address at risk.

58

Local vs. Global Blocks• If we decide to block a site, that block could

be strictly local (e.g., a file on just onesystem or site), or we can share that filterwith a few friends, or we might submit asite that merits being blocked to one ormore widely available DNS blacklists.

• Which is "better" or "worse" (from the pointof view of the sysadmin of a listed system)?To be blocked system-by-system? Or to beblocked at a whole bunch of sites due tobeing listed in a national blacklist?

59

Widely Used Blacklists• Being blocked by a widely used blacklist is

quite unpleasant (from the point of view ofthe system administrator who's subject tothat block) because suddenly a materialchunk of the Internet refuses his traffic. Onthe other hand, being blocked via a singlewidely used blacklist is actually GOOD inthat if he can get his system secured anddelisted, his access to ALL those systemswill equally suddenly be restored to normal.

60

Death Of A Thousand Local Cuts• Contrast that with locally maintained filters.

If a site simply drops his traffic (rather thanbouncing it back to him), he may not evenknow that my mail is being filtered! If he's"lucky" and does learn that a site is filteringhis mail, and he can persuade them toreverse that filter, that will fix that one site,but doesn't fix any of the other sites thatmay also be locally filtering him. Once anaddress becomes widely locally filtered itmay never be fully usable again.

61

Spammers and theIP address "shortage"

• In a spammer's fantasy world (or a worldwhere IPv6 has been widely deployed), thesupply of IP addresses would be effectivelylimitless, and a spammer could use eachaddress only once if he wanted to, and thenmove on to a new address in someunpredictable pattern to prevent anticipatoryblocking. In most cases, however, it is rarefor a spammer to have even a /24 (256 IPaddresses) available for his direct use.

62

Thus, Spammers' Needfor Address Gyrations

• Having comprehended that, you nowunderstand why spammers go through suchwild gyrations to launch spam at you fromweird delivery channels. 1) Normal emaildirect from them gets blocked; 2) they can'treadily move to new, as-of-yet unblocked,address space; and 3) they still desperatelywant to share important news about low,low, low, mortgage rates with you. Theyneed to become "clever" to email you.

63

Spam Delivery Channels• There really aren't all that many ways that

spammers can send email to you:

1) Spammers can try to send you spam viaa direct connection, which will work wellonly until you filter those addresses. Thesesort of connections are the highest trafficvolume spam sources you'll see, and afavorite of spammers, but comparativelyeasy to block for the reasons we justdiscussed.

64

Spam Delivery Channels (2)• 2) Spammers can send you spam via a

“throw away” dialup modem account. Ingeneral, direct-from-dialup spam is quiteeasy to block (as a category), and is nowharder for spammers to use than it used tobe due to ISP caller ID capabilities andcredit card registration requirements.Dialups also aren't particularly good forlarge volume spamming due to the lowspeed of 56Kbps modems.

65

Spam Delivery Channels (3)• 3) Spammers can send spam directly from a

“throw away” web email account, such asthose offered by any of hundreds ofdifferent sources. (see, for example, the listsat http://www.emailaddresses.com/)Most free email providers have becomequite proficient at blocking large scaleabuse of their service, and the ones thataren't responsible are small and generallytrivial to filter. (Much spam may have a freeweb email "From" but not originate there)

66

An Aside About RFC2142• One prime indicator of whether or not a

provider is responsible is the existence ofabuse@<domain>, the abuse reportingaddress required by RFC2142.

• If you're not sure if an ISP has an abuseaddress and reads complaints sent to it, see:http://www.rfc-ignorant.org/

• http://www.abuse.net lists wacky variantabuse addresses used by some providers

• Does YOUR campus have an abuse@address? Are you on rfc-ignorant.org ?

67

Spam Delivery Channels (4)• 4) Spam sent via an open SMTP relay -- this

is potentially serious, but is easily identifiedand blocked. This was really the firstserious technological thrust from spammers,and was once a far more serious problemthan it is now (although it is still pops up,e.g., http://docs.info.apple.com/article.html?Artnum=106763 )5) Spam sent via an exploitable web cgi-binformmail script (not much of a problem, butoccasionally you'll see this one pop up, too)

68

Spam Delivery Channels (5)• 6) Spam sent via insecure network access

points (e.g., spam sent from a lab thatdoesn't require authentication, open ethernetjacks or insecure wireless hubs). Potentiallya very serious problem, and one that highereducation is somewhat notorious forignoring. How many of us have live jackswith full, anonymous Internet access viaunlocked classrooms, libraries, etc.?Sometimes we secure wireless, but notwired ports, etc. -- they ALL need security!

69

Spam Delivery Channels (6)• 7) Spam sent via open proxy servers

(currently the most diverse and mostserious spam source).

This category includes the new "make-new-open-proxies-for-spammers-to-use" virusessuch as Jeem and SoBig.

70

The Mechanics of Blocking• The mechanics of blocking all these spam

sources, once you understand what spamchannels exist, and that there are blacklistsdevoted to blocking them, is relativelystraightforward.

• This is the part all the geeks in the audiencehave been waiting for. :-)

71

Blocking Spamhausen• The most convenient and elegant way to

block most major spam gangs who aresending spam directly to you is through useof the SBL (Spamhaus Block List) fromhttp://www.spamhaus.org/sbl/

The SBL is free and can be used withsendmail or most other major mail transferagents. Zone transfers can be arranged forlarge sites making 400K+ queries/day.

72

The SBL is NOT Spews• A more aggressive/controversial approach

to blocking known spam sources (which Idon't recommend for most colleges anduniversities) would be to use Spews. Spewsis not related to the SBL. Spews attempts topersuade spam-tolerant ISPs to beresponsible by using progressively widerblacklist entries. For info on Spews, pleasesee: http://www.spews.org/

73

The mail-abuse.org RBL+• While the SBL is very good at covering

what it says it will cover, it doesn't attemptto cover all the various spam channelsspammers try to use. Thus, to block direct-from-dialup spam, spam sent via openSMTP relays, spam sent from some openproxies, and some additional particularlyegregious spam tolerant sites, you'll want touse the mail-abuse.org RBL+ See:http://www.mail-abuse.org/rbl+/

74

The RBL+ Isn't Free(But It Is Cheap for .edu's)

• Not-for-profit and educational sites canlicense use of the RBL+ in zone transfermode for $125/name server/year plus$5/thousand users. This translates to about$250/year for a university the size of theUniversity of Oregon. Query mode is alsoavailable, but priced so as to discourage itsuse by large sites. The costs for queryaccess is $150/name server (including 1000users), with additional users $75 per 500.

75

The Particular ProblemOf Open Proxy Servers

• While the RBL+ recently began to list openproxy servers, the open proxy problem iswidespread enough (300,000+ known openproxy servers at this time), and so popularwith spammers that it merits its ownsupplemental open proxy DNS blacklist.There are a number of open proxy DNSBL'savailable, but after considering everything,I'd recommend that you use the Easynet.nl(formerly Wirehub.nl) open proxy DNSBL.

76

Easynet.nl Open Proxy DNSBL• For information about the Easynet.nl/

Wirehub.nl blackhole list, please see:http://basic.wirehub.nl/blackholes.htmlNote that two different Wirehub blacklistsare available; the one mentioned aboveblocks many different spam sources; ifyou ONLY want to block open proxies, see:http://abuse.easynet.nl/proxies.html

• The Wirehub/Easynet list is free; zonetransfers are available via rsync or wget.

77

Learning More AboutOpen Proxies

• Open proxies are a fascinating topic in theirown right, and one you really should learnmore about (as proof of their importanceand currency, they were recently discussedin the New York Times, front page, abovethe fold: http://www.nytimes.com/2003/05/20/technology/20SPAM.html)

• See my presentation on them this April's I2Member Meeting in Arlington VA:http://darkwing.uoregon.edu/~joe/proxies/

78

Locally Maintained Filters AsAn Adjunct to Blacklists

• Even with all these blacklists, you may stillneed to augment these DNSBLs with locallymaintained filters.

• If you use sendmail, you will probablyimplement these local filters via/etc/mail/access with entries being eitherproblematic domains or IP address blocks. Besure to use sendmail's delay_checks option

• Note: these files can become large. Revisioncontrol systems (like RCS) are a good idea.

79

Dealing With Cable Modemand DSL Customers

• While the RBL+ includes the DUL (dialupuser list), which blocks mail sent directlyfrom a dialup user's system (while allowingthose users to send mail through their ISP'sSMTP server), it doesn't deal with mail sentdirectly by cable modem and DSLcustomers, who should also only be sendingemail through their ISP's SMTP server.This is an example where locallymaintained filters can really help out.

80

Helpful ISP DNS Conventions• Many cable modem and DSL providers

assign IP addresses to their cable modemand DSL customers that are easy to spot(such as addresses with a pattern like:<foo>.dsl.telesp.net.br). Having identifiedaddresses of that sort it is easy to add a setof rules to /etc/mail/access which will blockdirect-from-DSL and direct-from-cable-modem mail. Just be sure you don't blockthe provider's SMTP servers!

81

DSL Customers and DNS• One more caution about this -- some DSL

folks seem particularly prone toward:1) registering a domain of their own andpointing it at their DSL connection, while2) failing to create a corresponding PTR(reverse DNS number-to-name) record, and3) failing to route their email through theirprovider's SMTP server.

• These guys get blocked when their dottedquad (still) resolves to <foo>.dsl.<bar>.comrather than the domain they're trying to use.

82

Are There DNSBLs WeShouldn’t Use?

• There are DNSBLs that filter based onanything and everything; I would NOTencourage you to use every DNSBL thatsomeone happens to offer. You reallyshould carefully investigate the criteria usedin putting hosts on and taking hosts off theDNSBL (among other things) since you'redelegating a tremendous amount ofauthority to the operators of the DNSBLsyou decide to use.

83

Examples of DNSBLs I Don't Use• There are some DNSBLs that are, in my

opinion, overly broad. For example, someblock all traffic from Brazil, or from China.See: http://www.blackholes.us/ orhttp://korea.services.net/ or http://www.okean.com/asianspamblocks.html orhttp://countries.nerd.dk/ orhttp://www.cluecentral.net/rbl/ ).

• You are better off blocking particularvulnerabilities, or even entire spam-friendlyISPs, rather than entire countries.

84

V. Teaching Your UsersCrucial Spam Reporting Skills

Your users are key intelligencesources in the war on spam. Your jobas their leader is to train them so thatthey are ready to report meaningfully.

85

You Will Not Block ALL Spam

• No matter how good your filters, somespam will still slip through. When it does,you want to know about it so you can usethat spam to help improve the blacklists youuse.

Once your users are properly trained,spammers won't be able to send spam tothem without "burning" the addresses theyused to do so.

86

Yes, You Really Do Want YourUsers To Send You Their Spam• Some of you who may already be drowning

in your own personal spam may considerthe idea that you want your users to sendyou their spam, too, to be, well, absurd.Trust me, it's not an insane idea. You NEEDyour users participation and cooperationbecause your spam may not look likeTHEIR spam, and besides, the sooner spamgets reported, the sooner it can get dealtwith. Before long, volume will become low.

87

What Do Users Need To Do?• The goal for your users is to get them to the

point where they can consistently:-- report only spam they receive (not viruses,not legitimate message traffic), which was-- sent directly to one of your spam-filteredsystems (not sent through some off sitemailing list, departmental hosts, Hotmail, etc.),-- to the right local reporting address, within-- a day or so of the time the spam was sent,-- forwarded with full/expanded headers (andwith the rest of the message body there, too).

88

Just Tell Us About YourSpam, Ma'am, Not Viruses

• Users sometimes have a hard time tellingspam apart from virus infested messages,and may try to report both. We're reallyonly interested in having spam reported,because (a) viruses aren't intentional, (b)we already "defang" any executable contentsent via email, (c) we site license NortonAntivirus for the desktop, and (d) when wecomplain to ISPs about viruses, thosereports seem to accomplish little or nothing.

89

Teaching Users To SpotDefanged Viruses

• If you defang executable attachments as wedo, those executable attachments will allhave a three part file name ending in .txt Ifyou can get users to look at the file name oftheir attachments, you're 99% of the waythere. [The other key (unrelated) part of thevirus picture for them to understand is thatKlez forges From: headers; have them see:http://www.wired.com/news/technology/0,1282,52174,00.html for more info.]

90

The Problem Of UsersReporting Legitimate Traffic

• Occasionally users will forget that theyhave requested email from a vendor about aparticular product, or a legitimate emailmay have a suspicious subject line and mayget reported by users wary of opening it.That sort of email obviously isn't spam, andshouldn't be reported, and for the most partdoesn't tend to be, although you must becareful when rare cases do arise.

91

We're Sorry You're GettingSpammed on Hotmail, But...

• If your users are like ours, many of themhave accounts on Yahoo or Hotmail or other3rd party web email systems that they use inaddition to their institutional accounts.That's fine, but there's nothing we can do tohelp with spam they get on those accounts,so please don't send it in to us. Likewise, ifusers are on a departmentally administeredhost, that's great, but again, there's nothingwe can do to fix spam problems there.

92

Spam Arriving Via Mailing Lists• A more common problem is spam that gets

delivered to our users via some mailing listthey're on that's hosted elsewhere.Assuming you use the approach we outlinein this talk, spam needs to get filtered by thesite hosting that mailing list; once the spamhas hit a mailing list, its too late for us to doanything about it. Users need to complain tothe site that's hosting the list, or convincethe mailing list owner to make the list aclosed or moderated list.

93

Using The Right LocalReporting Address

• While you may be tempted to have localusers just report spam they receive topostmaster@<domain> or abuse@<domain>you really should consider creating a specialspam reporting address (we'd suggestspam@<domain>) so that spam processingcan be kept separate from other postmasteror abuse-related duties. (You should alsoavoid having users report their spam to yourown personal email address.)

94

Getting Your Spam To UsWhile It Is Still Fresh

• Users need to understand that spam needs tobe reported with a day or so of the time itwas sent. Partially this is a matter of dealingwith current issues (rather than ancienthistory that's already been dealt with), andpartially this is a practical issue associatedwith some reporting services such asSpamcop (which needs you to send reportswithin 72 hours). Three day weekends andvacations are the biggest problem here...

95

Forward The Spam,Don't Use "Bounce"

• Make sure your users know to use theforward command to send you spam theyreceive, rather than "bouncing" it to you.

• Why? Forward preserves the integrity of theReceived: headers, while bounce tends tocomingle the original headers with theheaders of the person bouncing the messageto you, making it hard to process and reportthat spam appropriately.

96

And Then We Come ToThe Issue Of Full Headers...

• Anyone who works on abuse handling/spammanagement will tell you that the biggestobstacle to users effectively reporting theirspam is getting them to enable full headers.

• My colleagues have built a nice set of how-to-enable full header pages for the emailclients that our users tend to use; you'rewelcome to use them as the basis for localhow-to-enable full header pages, too. Seehttp://micro.uoregon.edu/fullheaders/

97

Providing Full Headers IsTedious From Some Programs

• If you look through those how-to-get-full-header web pages, you'll see that getting fullheaders from some email products (such asMS Outlook/Outlook Express) can be verytedious, while in other cases it is a matter ofpushing one button. If the email programyou or your users use makes it hard toreport full headers, complain to that vendorso that enabling full headers can be handledcleanly in future releases of that product.

98

What About End-UserTraining Beyond That?

• It can be tempting to just teach end usershow to use Spamcop, rather than "staying inthe loop" and processing the spam theyreport centrally. There's nothing wrong withteaching users to report spam themselvesthis way, but resist the urge to do it just soyou don't have to see the spam that's stillcoming in. You need to be aware of what'sstill coming in, and you should feel pain ifyour users are getting hit with lots of spam.

99

End Users & Other Spam Tools• Some users (technical enthusiasts/geeks

for the most part) may also be interested inexperimenting with other anti-spam tools,such as running Spamassassin.

• I would encourage you to be tolerant andencouraging toward those folks, assumingthey don't become a support burden orgenerate system load issues (even if youwouldn't use the particular solution they'reexperimenting with system-wide).

100

What Do I Do With SpamAfter Users Send It In To Me?

• You may want to usehttp://www.spamcop.net/ to report the spamto the correct providers.

• If you subscribe to the RBL+, be sure toalso report any open proxies or open relaysyou discover to them.

• You may want to tweak local filters• You also can report illegal activities directly

to appropriate authorities.

101

Oh Yes: There's Also TheIssue Of User Socialization

• Beyond technical spam reporting, the otherthing that you really should be doing is"socializing" your email users. By this, Imean your users need to understand:-- not everyone reads their email via a webbrowser; politeness implies that plain text(not html) is the correct way to go-- sending a 20MB attachment isn't some-thing that all correspondents love getting,nor does "everyone" use Word/Excel/etc.

102

User Socialization (2)• -- "Vacation" auto responders are almost

always a bad idea and are seldom needed-- Sig files should be brief, if used at all-- Just because you have the technicalability (or the political clout) to send emailto everyone on campus doesn't mean thatyou should ("intraspam" can be a realproblem at some campuses)

• Helping your local users develop a cultureof responsible email usage is part of gettingmail back to "the way it used to be…"

103

The Importance Of UsersHaving Healthy Skepticism

• The other thing you need to inculcate inyour users is a sense of healthy skepticism:-- No, there isn't millions of dollars waitingto be shared with you in Nigeria. Really.-- No, our support staff would never ask youto mail your password to a random Yahooemail account, I promise.-- No, the jdbgmgr.exe "teddy bear" iconisn't a virus, no matter what that chain letterfrom your cousin told you; do not delete it.

104

Grizzled Veterans Survive TheStress Of The Spam War Well• The process of helping users become

somewhat worldly and healthily skeptical isalso an important component of preparingthem to wage war on spam. Fighting spamcan require a somewhat thick skin as youdeal with disgusting message topics, and ahigh level of motivation as you combat anunseen and constantly morphing enemy.Skeptical/cynical "battle hardened" usersare well equipped to meet those challenges.

105

Is There Anything We DON'TWant Our Users To Do?

• Yes. For example, we don't want them totake direct retaliatory action since they mayend up mailbombing or ping flooding aninnocent party who is being "Joe jobbed."

• We don't want our users to munge theiraddress (doesn't work, can cause all sorts ofsupport issues if done ineptly).

• We don't want users to just give up.• We don't want users to try to intentionally

solicit more spam "just for us to block." :-)

106

Thanks For The ChanceTo Speak To You Today!

• Are there any questions?