Blackhole DocumentationRelease 1.6.0

Kura

April 30, 2013

CONTENTS

1 Using the blackhole.io service 31.1 Testing via telnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Testing SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Getting the source code 5

3 Running your own server 73.1 Python versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Third party libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 Getting started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Tests & Coverage 134.1 Running tests manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Third party CI/Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 Blackhole + PyPy 15

6 FAQ 17

7 Reference 197.1 blackhole.connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197.2 blackhole.data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197.3 blackhole.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.4 blackhole.opts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.5 blackhole.ssl_utils . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.6 blackhole.state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217.7 blackhole.utils . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

8 Indices and tables 23

Python Module Index 25

i

ii

Blackhole Documentation, Release 1.6.0

.o8 oooo oooo oooo oooo o8o"888 ‘888 ‘888 ‘888 ‘888 ‘"’888oooo. 888 .oooo. .ooooo. 888 oooo 888 .oo. .ooooo. 888 .ooooo. oooo .ooooo.d88’ ‘88b 888 ‘P )88b d88’ ‘"Y8 888 .8P’ 888P"Y88b d88’ ‘88b 888 d88’ ‘88b ‘888 d88’ ‘88b888 888 888 .oP"888 888 888888. 888 888 888 888 888 888ooo888 888 888 888888 888 888 d8( 888 888 .o8 888 ‘88b. 888 888 888 888 888 888 .o .o. 888 888 888‘Y8bod8P’ o888o ‘Y888""8o ‘Y8bod8P’ o888o o888o o888o o888o ‘Y8bod8P’ o888o ‘Y8bod8P’ Y8P o888o ‘Y8bod8P’

Blackhole is a Tornado powered MTA (mail transport agent) that is designed for handling large volumes of emailwithout handling any of the messages and doing no disk bound I/O.

Blackhole is designed mostly for testing purposes and can be used to test numerous things suchs as;

• Email send rates, if you need to test how much mail you can send per minute, hour etc

• Email integration testing and finally

• if you work in the real world, chances are you’ll need work on a copy of production data from time to time. Youcan try to anonymous all the data but there is always a chance you’ll miss something. Configuring blackholeas your applications default SMTP gateway will remove any chance of a real person receiving an email theyshouldn’t have received.

CONTENTS 1

Blackhole Documentation, Release 1.6.0

2 CONTENTS

CHAPTER

ONE

USING THE BLACKHOLE.IO SERVICE

All data sent to blackhole.io will be forgotten instantly, we store nothing you send.

1. Point your application’s outgoing SMTP server to ‘blackhole.io’,

2. Sit back and watch mail never get delivered to a real user.

or, send an email to blackhole.io using an @blackhole.io address, any address is fine e.g.:

user1@blackhole.io

1.1 Testing via telnet

$ telnet blackhole 25Trying 198.199.126.159...Connected to blackhole.io.Escape character is ’^]’.220 2.2.0 OK, readyHELO fake.mail.server250 2.5.0 OK, doneMAIL FROM:<user@address.tld>250 2.5.0 OK, doneRCPT TO:<someone@another.tld>250 2.5.0 OK, doneDATA354 3.5.4 Start mail input; end with <CRLF>.<CRLF>To: Someone <someone@another.tld>From: User <user@address.tld>Subject: Bye

Bye bye email.251 2.5.1 OK, user not local, will forwardQUIT221 2.2.1 Thank you for speaking to meConnection closed by foreign host.

1.2 Testing SSL

3

Blackhole Documentation, Release 1.6.0

$ openssl s_client -connect blackhole.io:465CONNECTED(00000003)depth=0 C = GB, ST = London, L = London, O = blackhole.io, OU = blackhole.io, CN = blackhole.io, emailAddress = kura@blackhole.io... snip ...---220 2.2.0 OK, readyHELO fake.mail.server250 2.5.0 OK, doneMAIL FROM:<user@address.tld>250 2.5.0 OK, doneRCPT TO:<someone@another.tld>250 2.5.0 OK, doneDATA354 3.5.4 Start mail input; end with <CRLF>.<CRLF>To: Someone <someone@another.tld>From: User <user@address.tld>Subject: Bye

Bye bye email.251 2.5.1 OK, user not local, will forwardQUIT221 2.2.1 Thank you for speaking to meDONE

4 Chapter 1. Using the blackhole.io service

CHAPTER

TWO

GETTING THE SOURCE CODE

The source code is available under the MIT license from GitHub.

5

Blackhole Documentation, Release 1.6.0

6 Chapter 2. Getting the source code

CHAPTER

THREE

RUNNING YOUR OWN SERVER

3.1 Python versions

Python 2.6Python 2.7Python 3.2Python 3.3PyPy 1.9 # see notes belowPyPy 2.0 # see notes below

Blackhole works on Python 2.6 and 2.7, it also works with PyPy (see Blackhole + PyPy section below).

3.2 Third party libraries

tornado>=2.2.1,<=3.1setproctitle>=1.1.6 # setproctitle 1.1.7 and above are required for all PyPy versionsdeiman>=0.1.4 # older version of Deiman will not work because of API changes

3.3 Getting started

3.3.1 Installation

Packaged

From PyPI

pip install blackhole

Or

easy_install blackhole

From GitHub

pip install -e git+git://github.com/kura/blackhole.git#egg=blackhole

7

Blackhole Documentation, Release 1.6.0

From source

Download the latest tarball from PyPI or GitHub. Unpack and run:

python setup.py install

3.3.2 Configuration options

Configuration options can be passed via the command line as below:

--workers=NUM Number of child workers to spawn (default: # of CPUs/cores + 1master)

--host=IP IP address to bind go (default: 0.0.0.0)

--port=PORT Port to listen for connections on (default: 25)

--pid=FILE File to write process information to (default: /tmp/blackhole.pid)

--log=FILE File to write logs to (not very verbose) (default: /tmp/blackhole.log)

--user=USER User to drop privs to during run time. (default: blackhole)

--group=GROUP Group to drop privs to during run time. (default: blackhole)

--mode=MODE Mode to run blackhole in (accept, bounce, random, unavailable, of-fline) (default: accept) - see Modes section

--debug=BOOL Will set the debug flag and causes all received and sent com-mand/email data to be logged. Will degrade performance with highthroughput (default: False)

SSL options

--ssl=BOOL Enabled or disable SSL, requires SSL compiled in to Python andOpenSSL. True or False (default: True)

--ssl_port=PORT Port to listen for SSL enabled connections (default: 465)

--ssl_key=PATH X509 SSL keyfile

--ssl_cert=PATH X509 SSL certificate file

You can also specify the –conf option to load configuration from a file:

--conf=FILE Config file to parse and use. Overrides command line args

For more information on the configuration file format read Configuration file example.

Deprecated options

The following options are deprecated and should not be used.

--ssl_ca_certs_dir=PATH Path to your operating system’s repository of certificates au-thorities (default: None)

3.3.3 Configuration file example

8 Chapter 3. Running your own server

Blackhole Documentation, Release 1.6.0

host="0.0.0.0"port=25pid="/tmp/blackhole.io"mode="offline"ssl_key=/etc/ssl/private/blackhole.io.keyssl_cert=/etc/ssl/certs/blackhole.io.crt

3.3.4 Debug Flag

After much debate and several emails from developers a debug flag has been added to blackhole.

You can enable debug using command line options Configuration options or via the configuration file Configurationfile example.

Enabling the debug flag will log all incoming and outgoing commands to the blackhole log file as well as all data sentafter a DATA command.

Each connection/email gets it’s own unique identifier, this indentifier uses only hexidecimal characters and is 10character in length, which means at some point the identifier will be reused.

Example debug output

2013-04-14 07:30:30,788 - Connection from ’127.0.0.1’2013-04-14 07:30:31,813 - [9AF260AB7F] RECV: HELO2013-04-14 07:30:31,814 - [9AF260AB7F] SEND: 250 2.5.0 OK, done2013-04-14 07:30:35,074 - [9AF260AB7F] RECV: MAIL FROM:<1@1.com>2013-04-14 07:30:35,075 - [9AF260AB7F] SEND: 250 2.5.0 OK, done2013-04-14 07:30:38,402 - [9AF260AB7F] RECV: RCPT TO:<2@2.com>2013-04-14 07:30:38,402 - [9AF260AB7F] SEND: 250 2.5.0 OK, done2013-04-14 07:30:40,573 - [9AF260AB7F] RECV: DATA2013-04-14 07:30:40,573 - [9AF260AB7F] SEND: 354 3.5.4 Start mail input; end with <CRLF>.<CRLF>2013-04-14 07:30:47,143 - [9AF260AB7F] RECV: From: <1@1.com>2013-04-14 07:30:52,181 - [9AF260AB7F] RECV: To: <2@2.com>2013-04-14 07:30:54,647 - [9AF260AB7F] RECV: Subject: Testing debug2013-04-14 07:30:56,550 - [9AF260AB7F] RECV: Hi 2@2.com2013-04-14 07:30:57,372 - [9AF260AB7F] RECV: This is a debug message2013-04-14 07:30:58,286 - [9AF260AB7F] RECV: .2013-04-14 07:30:58,287 - [9AF260AB7F] SEND: 250 2.5.0 OK, done2013-04-14 07:31:04,023 - [9AF260AB7F] RECV: QUIT2013-04-14 07:31:04,024 - [9AF260AB7F] SEND: 221 2.2.1 Thank you for speaking to me2013-04-14 07:31:04,024 - Closing

3.3.5 Controlling the server

The following commands are for use when using blackhole without an init script.

blackhole start Starts the serverblackhole stop Stops the serverblackhole restart Restarts the serverblackhole status Shows the status of the server, running, not running etc

For more information on controlling the server using init.d/rc.d read Controlling the server with init.d/rc.d.

3.3. Getting started 9

Blackhole Documentation, Release 1.6.0

3.3.6 Controlling the server with init.d/rc.d

/etc/init.d/blackhole start Starts the server/etc/init.d/blackhole stop Stops the server/etc/init.d/blackhole restart Restarts the server/etc/init.d/blackhole status Shows the status of the server, running, not running etc

Installating the init.d/rc.d scripts

The init script depends on /etc/blackhole.conf being in place and configured.

Blackhole comes with a script that works with init.d/rc.d, to install it copy it from the init.d/YOUR_DISTRO folder inthe root directory of this project to /etc/init.d/.

The init scripts can be found here.

i.e. for Debian/Ubuntu users, mv the file from init.d/debian-ubuntu/ to /etc/init.d/.

Then make sure it’s executable:

chmod +x /etc/init.d/blackhole

To make blackhole start on a reboot use the following:

update-rc.d blackhole defaults

3.3.7 Modes

The server can run in several different modes, these are outlined below.

See the Response codes section for more information on responses and which mode responds with which codes.

3.3.8 accept

Accept all email with code 250, 251, 252 or 253.

3.3.9 bounce

Bounce all email with a random code, excluding 250, 251, 252, 253.

3.3.10 random

Randomly accept or bounce all email with a random code.

3.3.11 unavailable

Server always respondes with code 421 - service is unavailable.

3.3.12 offline

Server always responds with code 521 - server does not accept mail.

10 Chapter 3. Running your own server

Blackhole Documentation, Release 1.6.0

3.3.13 Response codes

All

Code message220 OK, ready221 Thank you for speaking to me250 OK, done251 OK, user not local, will forward252 OK, cannot VRFY user but will attempt delivery253 OK, messages pending354 Start mail input; end with <CRLF>.<CRLF>355 Octet-offset is the transaction offset421 Service not available, closing transmission channel450 Requested mail action not taken: mailbox unavailable451 Requested action aborted: local error in processing452 Requested action not taken: insufficient system storage454 TLS not available due to temporary reason458 Unable to queue message459 Not allowed: unknown reason500 Command not recognized501 Syntax error, no parameters allowed502 Command not implemented503 Bad sequence of commands504 Command parameter not implemented521 Machine does not accept mail530 Must issue a STARTTLS command first534 Authentication mechanism is too weak538 Encryption required for requested authentication mechanism550 Requested action not taken: mailbox unavailable551 User not local552 Requested mail action aborted: exceeded storage allocation553 Requested action not taken: mailbox name not allowed554 Transaction failed571 Blocked

Accept

This mode will respond with the following codes:

Codes250 251 252 253

Bounce

This mode will respond with the following codes:

Codes421 431 450 451 452 454 458 459521 534 550 551 552 553 554 571

3.3. Getting started 11

Blackhole Documentation, Release 1.6.0

Offline

This mode will respond with the following codes:

Codes521

Unavailable

This mode will respond with the following codes:

Codes421

12 Chapter 3. Running your own server

CHAPTER

FOUR

TESTS & COVERAGE

4.1 Running tests manually

Running tests manually is pretty simple, there is a Make target dedicated to it.

The test suite relies on unittest2 and nose, both if which get installed by the Make target during test running.

make tests

There is also a Make target for generating coverage:

make coverage

4.2 Third party CI/Coverage

You can find the latest build status on travis.

And the test coverage report on coveralls.

13

Blackhole Documentation, Release 1.6.0

14 Chapter 4. Tests & Coverage

CHAPTER

FIVE

BLACKHOLE + PYPY

PyPy is a Python interpreter and just-in-time compiler. PyPy focuses on speed, efficiency and compatibility with theoriginal CPython interpreter.

Blackhole works well under PyPy 1.9, 2.0 beta1 and 2.0 beta2, you can see performance improvements of up to 30%in certain situations.

However, blackhole does have issues with both PyPy 1.9 and 2.0 beta1 and 2.0 beta2 when using the pre-compiledbinaries, this is due to a conflict in the version of OpenSSL compiled in to PyPy and the version compiled in to yourCPython installation. If you wish to use blackhole with SSL support on PyPy I suggest you either compile PyPyyourself or try to make sure your PyPy and CPython have the same versions.

15

Blackhole Documentation, Release 1.6.0

16 Chapter 5. Blackhole + PyPy

CHAPTER

SIX

FAQ

A few people have emailed me questions about why blackhole exists, how I use it, why Tornado and things like thatso I have outlined some questions and responses in an faq.

17

Blackhole Documentation, Release 1.6.0

18 Chapter 6. FAQ

CHAPTER

SEVEN

REFERENCE

7.1 blackhole.connection

Provide mechanisms for processing socket data.

This module provides methods for Blackhole to use internally for binding and listening on sockets as well as processall incoming socket data and responding appropriately.

7.1.1 blackhole.connection

blackhole.connection.sockets()Spawn a looper which loops over socket data and creates the sockets.

It should only ever loop over a maximum of two - standard (std) and SSL (ssl).

This way we’re able to detect incoming connection vectors and handle them accordingly.

A dictionary of sockets is then returned to later be added to the IOLoop.

blackhole.connection.connection_stream(connection)Detect which socket the connection is being made on, create and iostream for the connection, wrapping it inSSL if connected over the SSL socket.

The parameter ‘connection’ is an instance of ‘socket’ from stdlib.

blackhole.connection.handle_command(line, mail_state)Handle each SMTP command as it’s sent to the server

The paramater ‘line’ is the currently stream of data ending in ‘n’. ‘mail_state’ is an instance of ‘black-hole.state.MailState’.

blackhole.connection.connection_ready(sock, fd, events)Accepts the socket connections and passes them off to be handled.

‘sock’ is an instance of ‘socket’. ‘fd’ is an open file descriptor for the current connection. ‘events’ is an integerof the number of events on the socket.

blackhole.connection.write_response(stream, mail_state, resp)Write the response back to the stream

7.2 blackhole.data

Provides SMTP response codes and methods for returning the correct response code.

19

Blackhole Documentation, Release 1.6.0

This module contains all usable SMTP response codes for returning through the socket. It also provides mechanismsfor picking response codes that mean a mail message has been accepted, rejected or that the server is offline.

7.2.1 blackhole.data

blackhole.data.response(response)Return an SMTP response code and message. ‘response’ an string refering to the code you wish to return.

blackhole.data.get_response()Get our response from available responses based on configuration settings.

blackhole.data.random_choice(response_list)Pick a random choice for the configured choices dictionary.

‘response_list’ is a list of available response types from ‘blackhole.data’, this can be: - AC-CEPT_RESPONSES - BOUCNE_RESPONSES - OFFLINE_RESPONSES - UNAVAILABLE_RESPONSES- RANDOM_RESPONSES

blackhole.data.response_message(response)Format our response in ESMTP format.

‘response’ is a string containing the reponse code you wish to return.

7.3 blackhole.log

Logging configuration for the blackhole server.

7.4 blackhole.opts

Command line options for the blackhole server.

Also creates a list of available ports for the server to be run on, based on configuration and responds with the helpmenu when requested or invalid options are given.

7.4.1 blackhole.opts

blackhole.opts.ports()Create and return a list of sockets we need to create.

A maximum of two will be returned, default is standard (std) and SSL (ssl) it is enabled.

blackhole.opts.print_help(file)Prints all the command line options to stdout.

7.5 blackhole.ssl_utils

Utility functions for SSL wrapped sockets.

This module provides a simple default SSL configuration for the blackhole server and also exposes a custom ‘Black-holeSSLException’ for SSL-based exceptions.

20 Chapter 7. Reference

Blackhole Documentation, Release 1.6.0

7.5.1 blackhole.ssl_utils

exception blackhole.ssl_utils.BlackholeSSLExceptionA simple Exception class

blackhole.ssl_utils.verify_ssl_opts()Verify our SSL configuration variables are correctly set-up.

7.6 blackhole.state

State object for the current connection.

7.6.1 blackhole.state

class blackhole.state.MailStateA state object used for remembering the current connections place in our runtime.

This is mostly used for figuring out if we’re receiving SMTP commands or have trigger the DATA command.

email_idEmail ID is used to assign commands sent and received against an email/connection ID.

Only utilized when debug flag is set.

readingMailState’s ‘reading’ property, used for figuring out where we are in the state chain.

7.7 blackhole.utils

A utility module used for methods and features that do not belong in their own module.

7.7.1 blackhole.utils

blackhole.utils.setgid()Change our existing group.

Used to drop from root privileges down to a less privileged group.

MUST be called BEFORE setuid, not after.

blackhole.utils.setuid()Change our existing user.

Used to drop from root privileges down to a less privileged user

MUST be called AFTER setgid, not before.

blackhole.utils.terminate(signum, frame)Terminate the parent process and send signals to shut down it’s children

Iterates over the child pids in the frame and sends the SIGTERM signal to shut them down.

blackhole.utils.set_process_title()Set the title of the process.

If the process is the master, set a master title, otherwise set worker.

7.6. blackhole.state 21

Blackhole Documentation, Release 1.6.0

blackhole.utils.email_id()Generate an HEX ID to assign to each connection.

Will be reused later down the line due to the limited number of characters.

22 Chapter 7. Reference

CHAPTER

EIGHT

INDICES AND TABLES

• genindex

• modindex

• search

23

Blackhole Documentation, Release 1.6.0

24 Chapter 8. Indices and tables

PYTHON MODULE INDEX

bblackhole.connection (Unix), 19blackhole.data (Unix), 19blackhole.log (Unix), 20blackhole.opts (Unix), 20blackhole.ssl_utils (Unix), 20blackhole.state (Unix), 21blackhole.utils (Unix), 21

25