Kamaelia Grey
-
Upload
kamaelian -
Category
Technology
-
view
3.751 -
download
2
Transcript of Kamaelia Grey
Michael Sparks , [email protected]
Kamaelia Grey
Relevant disclosure?- Kamaelia came out of my day work @ BBC Research- Kamaelia grey came out of my personal frustration, and has nothing to do with the BBC :-)
Drastic Spam reduction through Greylisting
Greylisting made easy through natural concurrency
Natural concurrency made easy through Kamaelia
... since that's what Kamaelia is for
Kamaelia Grey
Problem?
Spam was getting ridiculous
One weekend the number of emails to home address topped 1000
96% was spam
4% wasn't, but was mainly mailing lists
Around 1% interesting
Next to impossible to find
NB: I run my own mail server :-)
Kamaelia
Day project at the BBC since 2003
Toolkit for making concurrency easy to work with in the context of
Networking (servers/clients)
Multimedia
TV
Pretty much anything
Designed to make writing servers fast, scalable & reliable
Greylisting 1
Client connects, wants to send us mail
Have we let their email through before?
YES!
I CAN HAZACCEPT ALL UR EMAIL
Greylisting
Client connects, wants to send us mail
Have we seen them before?
NO!
KTHXBYE
Greylisting
Client comes back after KTHXBYE
Is it a resend of the same message after a reasonable delay?
Greylisting
Client comes back after KTHXBYE
Is it a resend of the same message after a reasonable delay?
YES!
I CAN HAZACCEPT ALL UR EMAIL
Greylisting
Client comes back after KTHXBYE
Is it a resend of the same message after a reasonable delay?
NO!
KTHXBYE
Where?
Here!
Requirements
Python 2.3 onwards
Must run your own mail server already
Requires Kamaelia, but the tar ball includes that.
I developed it under Linux
I've got it deployed on Mac OS X
Mail serving
UNTRUSTEDINTERNET
Mail serving
UNTRUSTEDINTERNETMAILSERVER25
Mail serving
UNTRUSTEDINTERNETMAILSERVER25
Mail serving
UNTRUSTEDINTERNETMAILSERVER
25
Greylisting
UNTRUSTEDINTERNETMAILSERVER
8025
Greylisting
UNTRUSTEDINTERNETMAILSERVER
8025KamGrey25
Greylisting
UNTRUSTEDINTERNETMAILSERVER
8025KamGrey25
Installing
Grab the code & install
python setup.py install
Configure
edit /etc/greylist.conf
then run
/etc/init.d/kamaeliagrey start
I actually just use
App/runGreylister.sh
Basic config
port = 25servername = mail.cerenity.orgserverid = Kamaelia-SMTP 1.0smtp_ip = 192.168.2.9smtp_port = 8025inactivity_timeout = 60greylist_log = /var/log/greylist.loggreylist_debuglog = /var/log/greylist-debug.logattempters_db = /var/spool/attempters.dbmgreylisted_db = /var/spool/greylisted.dbm
Local config
allowed_senders: 127.0.0.1
allowed_sender_nets: 192.168.2
allowed_domains: private.thwackety.com thwackety.com thwackety.net yeoldeclue.com michaelsparks.info lansdowneresidents.org polinasparks.com pixienest.com kamaelia.org owiki.org cerenity.org
Exceptions
# Standard triples which always allows delivery to domains we consider# accepting for.## Format:# IP address, mail-from id, recipient id#whitelisted_triples: 213.38.186.202
Exceptions
# Some non-standard triples for which we always allow delivery# to domains we consider accepting for.# Format:# claimed sender name, IP prefix, recipient#whitelisted_nonstandard_triples: listmail.artsfb.org.uk 62.73.155.19 mx-out.facebook.com 204.15.20 mx-out.facebook.com 204.15.20 fallbackmx-out.facebook.com 204.15.20 fallbackmx-out.facebook.com 204.15.20
Logging?
Standard log:/var/log/greylist.log
20071106172215.940 | dsl88.241-20782.ttnet.net.tr | 88.241.81.46
| | | DEFERRED |
20071106172231.269 | dsl88.241-20782.ttnet.net.tr | 88.241.81.46 |
| | DEFERRED |20071106172238.610 | 34-148.privatnet.cz |
88.146.148.34 | | | DEFERRED |20071106172244.333 |
dsl88.241-20782.ttnet.net.tr | 88.241.81.46 | | | DEFERRED
|20071106172247.885 | mta410.k.cheetahmail.com | 208.49.63.136 | |
| ACCEPTED |
Logging?
Debug log:/var/log/greylist.log250 OK250 ACCEPTED451 4.7.1 Please try again later220 mail.cerenity.org ESMTP Kamaelia-SMTP 1.0 Tue Nov 6 17:26:01 2007500 Command Not Recognised250 mail.cerenity.org Hello lse 89.252.24.7250 OKNote this is what's being sent over SMTP
But also...
Debug log:/var/log/greylist-debug.log
*debug* THREADS['Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.Selector.Selector_11', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6']
Note this says the current internal components running
Internals
Debug log:/var/log/greylist-debug.log
['Kamaelia.Chassis.Pipeline.Pipeline_7',
'Kamaelia.Internet.Selector.Selector_11',
'__main__.GreylistServer_8',
'__main__.PeriodicWakeup_5',
'__main__.TCPS_10',
'__main__.WakeableIntrospector_6']
Note this says the current internal components running
Internals
Debug log:/var/log/greylist-debug.log
Kamaelia.Internet.Selector.Selector_11 wakes system on network
events
__main__.TCPS_10 sits inside and handles listening for
'__main__.GreylistServer_8 A configured network server
Kamaelia.Chassis.Pipeline.Pipeline_7 For
debugging'__main__.PeriodicWakeup_5 For debugging
'__main__.WakeableIntrospector_6 For debugging
Note this says the current internal components running
No connection!
TCPServerGreylistServerSelector
New connection!
TCPServerGreylistServerSelectorConnectedSocketAdapterGreyListingPolicy
More connections!
TCPServerGreylistServerSelectorConnectedSocketAdapterGreyListingPolicy
ConnectedSocketAdapterGreyListingPolicy
ConnectedSocketAdapterGreyListingPolicy
ConnectedSocketAdapterGreyListingPolicy
Accepted!
GreyListingPolicyTCPClientMAILSERVER8025
ConnectedSocketAdapter
Rejected!
ConnectedSocketAdapterGreyListingPolicy
Greylisting
GreyListingPolicyConcreteMailHandlerMailHandlerISAISA
Internals
The following pages are intended as a walk through of the key highlights of the code-base. It skips the body of the code since that's best looked at by scrolling through code and chatting rather than dumping into slides.
In a presentation its easy to say this, and then after going through slides giving road pointers go through the code.
Presentation files are a little more limited.
The code referred to is here: http://tinyurl.com/2sbjxl
Internals
class MailHandler(Axon.Component.component): logfile = "greylist.log" debuglogfile = "greylist-debug.log"
Internals
class MailHandler(Axon.Component.component): def __init__(self,**argd): def logging_recv_connection(self): def getline(self): def handleCommand(self,command): def noteToLog(self, line): def noteToDebugLog(self, line): def netPrint(self, *args): def lastline(self): def main(self):
Internals
class ConcreteMailHandler(MailHandler): Inboxes = { "inbox" : "Data from the client connecting to the server comes in here", "control" : "Shutdown & control messages regarding client side socket handling", "tcp_inbox" : "This is where we get respones from the real SMTP server", "tcp_control" : "This is where we get shutdown information from the real SMTP server", } Outboxes = { "outbox" : "Data sent here goes back the the client connecting to the server", "signal" : "Shutdown & control messages regarding client side socket handling", "tcp_outbox" : "Data sent here is sent to the real SMTP server", "tcp_signal" : "We send messages here to shutdown the connection to the real SMTP connection", } peer = "*** UNDEFINED ***" peerport = "*** UNDEFINED ***" local = "*** UNDEFINED ***" localport = "*** UNDEFINED ***" servername = "Testing.server.local" serverid = "MPS SMTP 1.0" smtp_ip = "192.168.2.9" smtp_port = 25
Internals
class ConcreteMailHandler(MailHandler): def connectToRealSMTPServer(self): def __init__(self, **argv): def error(self, message): def RelayError(self): def handleConnect(self): def handleEhlo(self,command): def handleHelo(self,command): def handleHelp(self,command): def handleVrfy(self,command): def handleRset(self,command): def handleNoop(self,command): def handleMail(self,command): def handleRcpt(self,command): def handleData(self, command): def handleQuit(self,command): def shouldWeAcceptMail(self): return False # Default policy - don't accept any email def deferMail(self): def acceptMail(self): def getline_fromsmtpserver(self): def handleDisconnect(self):
Internals
class GreyListingPolicy(ConcreteMailHandler): allowed_senders = [] allowed_sender_nets = [] allowed_domains = [ ]
Internals
class GreyListingPolicy(ConcreteMailHandler): (continued) def shouldWeAcceptMail(self): def sentFromAllowedIPAddress(self): def sentFromAllowedNetwork(self): def sentToADomainWeForwardFor(self): def isGreylisted(self, recipient): def whiteListed(self, recipient): def logResult(self):
Internals
class GreylistServer(MoreComplexServer): logfile =
config["greylist_log"] debuglogfile = config["greylist_debuglog"]
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) port =
config["port"] class TCPS(TCPServer): CSA =
NoActivityTimeout(ConnectedSocketAdapter,
timeout=config["inactivity_timeout"], debug=False)....
Internals
class GreylistServer(MoreComplexServer): (continued)# ... class
protocol(GreyListingPolicy): servername = config["servername"]
serverid = config["serverid"] smtp_ip = config["smtp_ip"] smtp_port
= config["smtp_port"] allowed_senders = config["allowed_senders"]
allowed_sender_nets = config["allowed_sender_nets"] allowed_domains
= config["allowed_domains"] whitelisted_triples =
config["whitelisted_triples"] whitelisted_nonstandard_triples
=
config["whitelisted_nonstandard_triples"]
And that...
Saves me having to wade through now literally tens of thousands of spams over the past two months:
# grep DEFERRED /var/log/greylist.log |wc -l 73798
Time to write?
2 days start to finish including protocol
Bug fixes ~1-2 days tops
Questions?
Thank you :-)I hope it's useful to you :)
Click to edit the title text format
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
Ninth Outline Level
http://kamaelia.sourceforge.net/KamaeliaGrey