heka-py Documentation

heka-py DocumentationRelease 0.30.3

Rob Miller

September 22, 2016

Contents

1 heka-py 31.1 Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Configuration formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Example 1: Use your framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Example 2: Module scope, if you must . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1.2 Heka Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5INI format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6HMAC signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.3 Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.1.4 Message Encoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

NullEncoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8StdlibPayloadEncoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8ProtobufEncoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.1.5 Output streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8DebugCaptureStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8FileStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8StdOutStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9StdLibLoggingStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9TcpStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9UdpStream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.1.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.1.7 Dictionary Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.1.8 Debugging your configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 API Documentation 112.1 Config API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 HekaClient API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3 Development Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Production Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.5 Encoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.5.1 ProtobufEncoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.5.2 StdlibPayloadEncoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.6 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.7 Decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.8 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Python Module Index 19

i

heka-py Documentation, Release 0.30.3

This project is no longer being maintained. A simplified version of the core of this library is available at the followingfork:

https://github.com/kalail/heka-py

Support for Heka is available on the Heka mailing list (https://mail.mozilla.org/listinfo/heka) or on the #heka channelon irc.mozilla.org.

Contents 1

https://github.com/kalail/heka-py

https://mail.mozilla.org/listinfo/heka


2 Contents

CHAPTER 1

heka-py

heka-py is a Python client for the “Heka” system of application logging and metrics gathering developed by the MozillaServices team. The Heka system is meant to make life easier for application developers with regard to generating andsending logging and analytics data to various destinations. It achieves this goal (we hope!) by separating the concernsof message generation from those of message delivery and analysis. Front end application code no longer has to dealdirectly with separate back end client libraries, or even know what back end data storage and processing tools are inuse. Instead, a message is labeled with a type (and possibly other metadata) and handed to the Heka system, whichthen handles ultimate message delivery.

A pre-rendered version of this documentation is available on heka-py.

1.1 Contents

1.1.1 Getting Started

There are two primary components with which users of the heka-py library should be aware. The first is the HekaClientAPI client class. The HekaClient exposes the Heka API, and is generally your main point of interaction with the Hekasystem. The client doesn’t do very much, however; it just provides convenience methods for constructing messagesof various types and then passes the messages along. Actual message delivery is handled by a encoding a messageobject into a wireformat using encoders: and then transported using streams. without a properly configured encoderand stream, a hekaclient is useless.

The first question you’re likely to ask when using heka-py, then, will probably be “How the heck do I get my hands ona properly configured client?” You could read the source and instantiate and configure these objects yourself, but foryour convenience we’ve provided a Heka Configuration module that simplifies this process considerably. The configmodule provides utility functions that allow you pass in a declarative representation of the settings you’d like for yourclient and stream objects, and it will create and configure them for you based on the provided specifications.

Configuration formats

As described in detail on the Heka Configuration page, you can create a HekaClient from either an INI-file style text configuration (using client_from_text_config or client_from_stream_config) or a dictionary (usingclient_from_dict_config). The text version is simply parsed and turned into a dictionary internally, so the followingtext config:

[heka]logger = myappseverity = 4disabled_timers = foo

3

https://wiki.mozilla.org/Services

https://wiki.mozilla.org/Services

http://heka-py.readthedocs.org


barstream_class = heka.streams.UdpStreamstream_host = localhoststream_port = 5565

Will be converted into this dictionary config:

{'disabled_timers': ['foo', 'bar'],'filters': [],'hmac': {},'logger': 'myapp','plugins': {},'severity': 4,'stream': {'class': 'heka.streams.UdpStream',

'host': 'localhost','port': 5565},

'stream_class':'heka.streams.UdpStream','stream_host': 'localhost','stream_port': 5565}

Let’s ignore the details of the config options for now (again, see Heka Configuration for the nitty-gritty) and justassume that you have a working client configuration. How might you actually use this within your code to generate aclient and make that client avaialable for use? A couple of mechanisms are described below.

Example 1: Use your framework

It is of course possible to simply define or load your configuration up at module scope and then use it to create a clientthat is then available to the rest of the module, like so:

from heka.config import client_from_stream_config

with open('/path/to/config.ini', 'r') as inifile:hekager = client_from_stream_config(inifile, 'heka')

hekager.heka('msgtype', payload='Message payload')

However, this is considered by many (including the Heka authors) to be A Bad Idea®. Instead of creating import timeside effects, you can use your application’s bootstrap code to initialize your client and make it available as needed.How exactly this should work varies from application to application. For instance, the Pyramid web framework expectsa main function to be defined at the root of your application’s package. This might contain the following code:

from heka.config import client_from_stream_configfrom pyramid.config import Configurator

def main(global_config, **settings):""" This function returns a Pyramid WSGI application."""config = Configurator(settings=settings)with open('/path/to/config.ini', 'r') as heka_ini:

heka_client = client_from_stream_config(heka_ini, 'heka')config.registry['heka'] = heka_client

config.add_static_view('static', 'static', cache_max_age=3600)config.add_route('home', '/')config.scan()return config.make_wsgi_app()

4 Chapter 1. heka-py

http://www.plope.com/Members/chrism/import_time_side_effects

http://www.pylonsproject.org/


Then the HekaClient instance will be available on Pyramid’s “registry”, which is always available through the requestobject or a library call. With a bit more code you could put your Heka configuration info into Pyramid’s default configfile and then extract it from the settings values passed in to the main function. The idea is that you make use ofwhatever infrastructure or patterns that your application and/or framework provide and cooperate with those to createand make available a client for logging and metrics-gathering needs.

Example 2: Module scope, if you must

Despite the fact that some consider it to be an anti-pattern, there are those who are quite fond of the import logging;logger = logging.getLogger(‘foo’) idiom that the stdlib logging package provides for making a logger available atmodule scope. We recommend that you consider not doing so and instead making your client available throughsome application- or framework-specific mechanism, but if you really want to stick to your guns then there’s a bit ofconvenience that heka-py provides.

The short version is that where you would have done this:

from logging import getLoggerlogger = getLogger('myapp')

Instead you’d do the following:

from heka.holder import get_clientheka_log = get_client('myapp')

Every time throughout your application’s process, a call to get_client(‘myapp’) will return the same HekaClient in-stance. At this point, however, the client in question is still not usable, because it doesn’t have a working stream. Again,the recommendation is that somewhere in your application code you use one of the config functions to initialize theclient, which might look like this:

from heka.config import client_from_stream_configfrom heka.holder import get_clientheka_log = get_client('myapp')

def some_init_function():with open('/path/to/heka.ini', 'r') as heka_ini:

client_from_stream_config(heka_ini, 'heka', heka_log)

Note that the heka_log client was passed in to the client_from_stream_config call, which causes the configuration tobe applied to that client rather than a new client being created.

If you really want to do all of your initialization at module scope, you can pass a config dict to the get_client function.This is a minimal working configuration that will cause all Heka output to be sent to stdout:

from heka.holder import get_clientheka_config = {'stream': {'class': 'heka.streams.StdOutStream'}}heka_log = get_client('myapp', heka_config)

1.1.2 Heka Configuration

To assist with getting a working Heka set up, heka-py provides a Config API module which will take declarativeconfiguration info in either ini file or python dictionary format and use it to configure a HekaClient instance. Even ifyou choose not to use these configuration helpers, this document provides a good overview of the configurable optionsprovided by default by the HekaClient API client class.

The config module will accept configuration data either in ini format (as a text value or a stream input) or as aPython dictionary value. This document will first describe the supported ini file format, followed by the correspondingdictionary format to which the ini format is ultimately converted behind the scenes.

1.1. Contents 5

http://www.plope.com/Members/chrism/logging_blues


INI format

The primary HekaClient configuration should be provided in a heka section of the provided ini file text. (Note thatthe actual name of the section is passed in to the config parsing function, so it can be any legal ini file section name,but for illustration purposes these documents will assume that the section name is heka.) A sample heka section mightlook like this:

[heka]logger = myappseverity = 4disabled_timers = foo

barstream_class = heka.streams.UdpStreamstream_host = localhoststream_port = 5565

Of all of these settings, only stream_class is strictly required. A detailed description of each option follows:

logger Each heka message that goes out contains a logger value, which is simply a string token meant to identify thesource of the message, usually the name of the application that is running. This can be specified separately foreach message that is sent, but the client supports a default value which will be used for all messages that don’texplicitly override. The logger config option specifies this default value. This value isn’t strictly required, but ifit is omitted ‘’ (i.e. the empty string) will be used, so it is strongly suggested that a value be set.

severity Similarly, each heka message specifies a severity value corresponding to the integer severity values definedby RFC 3164. And, again, while each message can set its own severity value, if one is omitted the client’s defaultvalue will be used. If no default is specified here, the default default (how meta!) will be 6, “Informational”.

disabled_timers Heka natively supports “timer” behavior, which will calculate the amount of elapsed time taken byan operation and send that data along as a message to the back end. Each timer has a string token identifier.Because the act of calculating code performance actually impacts code performance, it is sometimes desirableto be able to activate and deactivate timers on a case by case basis. The disabled_timers value specifies a setof timer ids for which the client should NOT actually generate messages. Heka will attempt to minimize therun-time impact of disabled timers, so the price paid for having deactivated timers will be very small. Note thatthe various timer ids should be newline separated.

stream_class This should be a Python dotted notation reference to a class (or factory function) for a Heka “stream”object. A stream needs to provide a write(data) method, which is responsible for accepting a byte serializedmessage and passing it along to the router / back end / output mechanism / etc. heka-py provides some develop-ment streams, but the main one it provides for intended production use makes use of UDP to send messages toany configured listeners.

stream_* (excluding stream_class) As you might guess, different types of streams can require different configura-tion values. Any config options other than stream_class that start with stream_ will be passed to the streamfactory as keyword arguments, where the argument name is the option name minus the stream_ component andthe value is the specified value. In the example above, the UDP host and port will be passed to the UdpStreamconstructor.

encoder: This should be a Python dotted notation reference to a class (or factory function) for a Heka “encoder”object. An encoder needs to provider a encode(msg) method which is responsible for serializing an instance ofheka.message.Message and returning a byte serialized version of the message. Currently, only a ProtobufEn-coder is supported and is the default.

In addition to the main heka section, any other config sections that start with heka_ (or whatever section name isspecified) will be considered to be related to the heka installation. Only specific variations of these are supported,however. The first of these is configuration for HekaClient Filters. Here is an example of such a configuration:

[heka_filter_sev_max]provider = heka.filters.severity_max_provider


https://www.ietf.org/rfc/rfc3164.txt


severity = 4

[heka_filter_type_whitelist]provider = heka.filters.type_whitelist_providertypes = timer

oldstyle

Each heka_filter_* section must contain a provider entry, which is a dotted name specifying a filter provider function.The rest of the options in that section will be converted into configuration parameters. The provider function will becalled and passed the configuration parameters, returning a filter function that will be added to the client’s filters. Thefilters will be applied in the order they are specified. In this case a “severity max” filter will be applied, so that onlymessages with a severity of 4 (i.e. “warning”) or lower will actually be passed in to the stream. Additionally a “typewhitelist” will be applied, so that only messages of type “timer” and “oldstyle” will be delivered.

HMAC signatures

Messages can be signed with an HMAc. You may use either SHA1 or MD5 to sign messages.

The INI configuration looks for a section that is the same as your client, but is post-fixed with “_hmac”.

An example configuration in INI format looks like

[heka]stream_class = heka.streams.DebugCaptureStream

[heka_hmac]signer = some_signer_namekey_version = 2hash_function = SHA1key = some_key_value

All HMAC signatures and metadata are stored in the Heka header to be decoded by a heka daemon.

1.1.3 Plugins

Heka allows you to bind new extensions onto the client through a plugin mechanism.

Each plugin must have a configuration section name with a prefix of heka_plugin_. Configuration is parsed into adictionary, passed into a configurator and then the resulting plugin method is bound to the client.

Each configuration section for a plugin must contain at least one option with the name provider. This is a dotted namefor a function which will be used to configure a plugin. The return value for the provider is a configured method whichwill then be bound into the Heka client.

Each plugin extension method has a canonical name that is bound to the heka client as a method name. The suffix thatfollows the heka_plugin_ prefix is used only to distinguish logical sections for each plugin within the configurationfile.

An example best demonstrates what can be expected. To load the dummy plugin, you need a heka_plugin_dummysection as well as some configuration parameters. Here’s an example

[heka_plugin_dummysection]provider=heka.tests.plugin.config_pluginport=8080host=localhost

Once you obtain a reference to a client, you can access the new method.

1.1. Contents 7


from heka.holder import CLIENT_HOLDERclient = CLIENT_HOLDER.get_client('your_app_name')client.dummy('some', 'ignored', 'arguments', 42)

1.1.4 Message Encoders

NullEncoder

This encoder passes protocol buffer objects through the encode() function. This is only used for debugging purposes

StdlibPayloadEncoder

The StdlibPayloadEncoder must be used in conjunction with the StdLibLoggingStream. This encoder is a lossy outputstream which only writes out the payload section to the Python logger.

ProtobufEncoder

The ProtobufEncoder writes messages using raw protocol buffers. Note that a small protocol buffer header is alsoprefixed to the message so that the hekad daemon can decode the message.

1.1.5 Output streams

All streams are visible under the heka.streams namespace.

DebugCaptureStream

This stream captures messages and stores them in a msgs queue. Note that the encoder you use may make it awkwardto read messages out of the queue. You can use the NullEncoder for testing purposes which will simply queue up theprotocol buffer objects for you.

Example config

[heka]stream_class = heka.streams.DebugCaptureStreamencoder = heka.encoders.ProtobufEncoder

FileStream

This stream appends messages to a file.

Example config

[heka]stream_class = heka.streams.DebugCaptureStream



StdOutStream

This stream captures messages and writes them to stdout.

Example config

[heka]stream_class = heka.streams.StdOutStream

StdLibLoggingStream

This stream captures messages and writes them to the python standard logger. Currently you must use the StdlibPay-loadEncoder with this output stream.

Example configuration

[heka]stream_class = heka.streams.StdLibLoggingStreamstream_logger_name = HekaLoggerencoder = heka.encoders.StdlibPayloadEncoder

TcpStream

The TcpStream writes messages to one or more hosts. There is currently minimal support for error handling if a socketis closed on the the remote host.

Example

[heka]stream_class = heka.streams.TcpStreamstream_host = 192.168.20.2stream_port = 5566

UdpStream

The UdpStream writes messages to one or more hosts.

Example

[heka]stream_class = heka.streams.UdpStreamstream_host = 192.168.20.2stream_port = 5565

1.1.6 Examples

Working examples are included in the examples directory in the git repository for you.

1.1.7 Dictionary Format

When using the client_from_text_config or client_from_stream_config functions of the config module to parsean ini format configuration, heka-py simply converts these values to a dictionary which is then passed toclient_from_dict_config. If you choose to not use the specified ini format, you can parse configuration yourself and

1.1. Contents 9


call client_from_dict_config directly. The configuration specified in the “ini format” section above would be convertedto the following dictionary:

{'logger': 'myapp','severity': 4,'disabled_timers': ['foo', 'bar'],'stream': {'class': 'heka.streams.UdpStream',

'host': 'localhost','port': 5565,

},'filters': [('heka.filters.severity_max',

{'severity': 4},),('heka.filters.type_whitelist',{'types': ['timer', 'oldstyle']},

),],

}

To manually load a Heka client with plugins, the client_from_dict_config function allows you to pass in a list of pluginconfigurations using the plugins dict key, used in the same fashion as filters in the example directly above.

The configuration specified in the “plugins” section above would be converted into the following dictionary, where thekey will be the name of the method bound to the client:

{'dummy': ('heka.tests.plugin:config_plugin',{'port': 8080,'host': 'localhost'

},)

}

1.1.8 Debugging your configuration

You may find yourself with a heka client which is not behaving in a manner that you expect. Heka provides a deepcopyof the configuration that was used when the client was instantiated for debugging purposes.

The following code shows how you can verify that the configuration used is actually what you expect it to be

import jsonfrom heka.config import client_from_dict_config

cfg = {'logger': 'addons-marketplace-dev','stream': {'class': 'heka.streams.UdpStream','host': ['logstash1', 'logstash2'],'port': '5566'}}

client = client_from_dict_config(cfg)assert client._config == json.dumps(cfg)


CHAPTER 2

API Documentation

2.1 Config API

Helpers to handle HekaClient configuration details.

heka.config._convert(value)Converts a config value. Numeric integer strings are converted to integer values. ‘True-ish’ string values areconverted to boolean True, ‘False-ish’ to boolean False. Any alphanumeric (plus underscore) value enclosedwithin ${dollar_sign_curly_braces} is assumed to represent an environment variable, and will be converted tothe corresponding value provided by os.environ.

heka.config.client_from_dict_config(config, client=None)Configure a heka client, fully configured w/ stream and plugins.

Parameters

• config – Configuration dictionary.

• client – HekaClient instance to configure. If None, one will be created.

The configuration dict supports the following values:

logger Heka client default logger value.

severity Heka client default severity value.

disabled_timers Sequence of string tokens identifying timers that are to be deactivated.

filters Sequence of 2-tuples (filter_provider, config). Each filter_provider is a dotted name referring to a func-tion which, when called and passed the associated config dict as kwargs, will return a usable HekaClientfilter function.

plugins Nested dictionary containing plugin configuration. Keys are the plugin names (i.e. the name themethod will be given when attached to the client). Values are 2-tuples (plugin_provider, config). Eachplugin_provider is a dotted name referring to a function which, when called and passed the associatedconfig, will return the usable plugin method.

stream Nested dictionary containing stream configuration.

All of the configuration values are optional, but failure to include a stream may result in a non-functional Hekaclient. Any unrecognized keys will be ignored.

Note that any top level config values starting with stream_ will be added to the stream config dictionary, over-writing any values that may already be set.

The stream configuration supports the following values:

11


class (required) Dotted name identifying the stream class to instantiate.

args Sequence of non-keyword args to pass to stream constructor.

<kwargs> All remaining key-value pairs in the stream config dict will be passed as keyword arguments to thestream constructor.

heka.config.client_from_stream_config(stream, section, client=None)Extract configuration data in INI format from a stream object (e.g. a file object) and use it to generate a Hekaclient. Config values will be sent through the _convert function for possible type conversion.

Parameters

• stream – Stream object containing config information.

• section – INI file section containing the configuration we care about.


Note that all stream config options should be prefaced by “stream_”, e.g. “stream_class” should specify thedotted name of the stream class to use. Similarly all extension method settings should be prefaced by “exten-sions_”.

heka.config.client_from_text_config(text, section, client=None)Extract configuration data in INI format from provided text and use it to configure a Heka client. Text isconverted to a stream and passed on to client_from_stream_config.

Parameters

• text – INI text containing config information.



heka.config.dict_from_stream_config(stream, section)Parses configuration from a stream and converts it to a dictionary suitable for passing to client_from_dict_config.

Parameters

• stream – Stream object containing config information.


heka.config.nest_prefixes(config_dict, prefixes=None, separator=’_’)Iterates through the config_dict keys, looking for any starting w/ one of a specific set of prefixes, moving thoseinto a single nested dictionary keyed by the prefix value.

Parameters

• config_dict – Dictionary to mutate. Will also be returned.

• prefixes – Sequence of prefixes to look for in config_dict keys.

• separator – String which separates prefix values from the rest of the key.

2.2 HekaClient API

class heka.client.HekaClient(stream, logger, severity=6, disabled_timers=None, filters=None, en-coder=’heka.encoders.ProtobufEncoder’, hmc=None)

Client class encapsulating heka API, and providing storage for default values for various heka call settings.

add_method(method, override=False)Add a custom method to the HekaClient instance.

12 Chapter 2. API Documentation


Parameters

• method – Callable that will be used as the method.

• override – Set this to the method name you want to override. False indicates no over-ride will occur.

critical(msg, *args, **kwargs)Log a CRITICAL level message

debug(msg, *args, **kwargs)Log a DEBUG level message

env_version = ‘0.8’

error(msg, *args, **kwargs)Log an ERROR level message

exception(msg, exc_info=True, *args, **kwargs)Log an ALERT level message

gauge(name, value, logger=None, severity=None, fields=None, rate=1.0)Sends an ‘current gauge measurement’ message.

Parameters

• name – String label for the gauge.

• value – Number current absolute value of the gauge.

• logger – String token identifying the message generator.

• severity – Numerical code (0-7) for msg severity, per RFC 5424.

• fields – Arbitrary key/value pairs for add’l metadata.

heka(type, logger=None, severity=None, payload=’‘, fields=None, timestamp=None)Create a single message and pass it to the sender for delivery.

Parameters

• type – String token identifying the type of message payload.



• payload – Actual message contents.


• timestamp – Custom timestamp for the message. If no timestamp is given, then currenttime will be used.

incr(name, count=1, logger=None, severity=None, fields=None, rate=1.0)Sends an ‘increment counter’ message.

Parameters

• name – String label for the counter.

• count – Integer amount by which to increment the counter.




2.2. HekaClient API 13


info(msg, *args, **kwargs)Log an INFO level message

is_active

send_message(msg)

setup(stream, encoder, hmc, logger=’‘, severity=6, disabled_timers=None, filters=None)Setup the HekaClient

Parameters

• logger – Default logger value for all sent messages.

• severity – Default severity value for all sent messages.

• disabled_timers – Sequence of string tokens identifying timers that should be deac-tivated.

• filters – A sequence of filter callables.

timer(name, logger=None, severity=None, fields=None, rate=1.0)Return a timer object that can be used as a context manager or a decorator, generating a heka ‘timer’message upon exit.

Parameters

• name – Required string label for the timer.




• rate – Sample rate, btn 0 & 1, inclusive (i.e. .5 = 50%). Sample rate is enforced in thismethod, i.e. if a sample rate is used then some percentage of the timers will do nothing.

timer_send(name, elapsed, logger=None, severity=None, fields=None, rate=1.0)Converts timing data into a heka message for delivery.

Parameters

• name – Required string label for the timer.

• elapsed – Elapsed time of the timed event, in ms.




• rate – Sample rate, btn 0 & 1, inclusive (i.e. .5 = 50%). Sample rate is NOT enforced inthis method, i.e. all messages will be sent through to heka, sample rate is purely informa-tional at this point.

warn(msg, *args, **kwargs)Log a WARN level message

class heka.client.SEVERITYPut a namespace around RFC 3164 syslog messages

ALERT = 1

CRITICAL = 2

DEBUG = 7



EMERGENCY = 0

ERROR = 3

INFORMATIONAL = 6

NOTICE = 5

WARNING = 4

2.3 Development Streams

class heka.streams.dev.DebugCaptureStream(**kwargs)Captures up to 100 heka messages in a circular buffer for inspection later.

This is only for DEBUGGING. Do not use this for anything except development.

__weakref__list of weak references to the object (if defined)

write(msg)Append object to the circular buffer.

class heka.streams.dev.FileStream(filepath)Emits messages to a filesystem file.


class heka.streams.dev.StdOutStreamThis is implemented as a class so that mocks can properly stub out sys.stdout


2.4 Production Streams

class heka.streams.udp.UdpStream(host, port)Sends heka messages out via a UDP socket.

__init__(host, port)Create UdpStream object.

Parameters

• host – A string or sequence of strings representing the hosts to which messages shouldbe delivered.

• port – An integer or sequence of integers representing the ports to which the messagesshould be delivered. Will be zipped w/ the provided hosts to generate host/port pairs. Ifthere are extra hosts, the last port in the sequence will be repeated for each extra host. Ifthere are extra ports they will be truncated and ignored.


write(data)Send bytes off to the heka listener(s).

Parameters data – bytes to send to the listener

2.3. Development Streams 15


class heka.streams.tcp.TcpStream(host, port)Sends heka messages out via a TCP socket.

__init__(host, port)Create TcpStream object.

Parameters

• host – A string or sequence of strings representing the hosts to which messages shouldbe delivered.

• port – An integer or sequence of integers representing the ports to which the messagesshould be delivered. Will be zipped w/ the provided hosts to generate host/port pairs. Ifthere are extra hosts, the last port in the sequence will be repeated for each extra host. Ifthere are extra ports they will be truncated and ignored.


write(data)Send bytes off to the heka listener(s).

Parameters data – bytes to send to the listener

2.5 Encoders

2.5.1 ProtobufEncoder

class heka.encoders.ProtobufEncoder(hmc=None)

2.5.2 StdlibPayloadEncoder

class heka.encoders.StdlibPayloadEncoder(hmc=None)If an incoming message does not have a ‘loglevel’ set, we just use a default of logging.INFO

2.6 Filters

Callables that provide message filters for the HekaClient.

Each provider accepts needed filter configuration values, and returns a filter function usable by a HekaClient instance.

Each filter accepts a single msg dictionary argument and returns a boolean value: True if a message should be delivered,False if a message should not be delivered. Note that the msg dictionary may be mutated by the filter.

heka.filters.severity_max_provider(severity)Filter if message severity is greater than specified severity.

heka.filters.type_blacklist_provider(types)Filter if message type is in the types value.

heka.filters.type_severity_max_provider(types)Filter if message type has specified maximum severity value and message severity is higher than this maximum.

Each keyword argument key should be a message type name, and each keyword argument value should be themaximum allowed severity for that message type.



heka.filters.type_whitelist_provider(types)Filter if message type is NOT in the types value.

2.7 Decorators

This module contains a Heka decorator base class and some additional helper code. The primary reason for theseabstractions is ‘deferred configuration’. Decorators are evaluated by Python at import time, but often the configurationneeded for a Heka client, which might negate (or change) the behavior of a Heka decorator, isn’t available until later,after some config parsing code has executed. This code provides a mechanism to have a function get wrapped in oneway (or not at all) when the decorator is originally evaluated, but then to be wrapped differently once the config hasloaded and the desired final behavior has been established.

class heka.decorators.base.HekaDecorator(*args, **kwargs)Base class for Heka decorators

Designed to support ‘rebinding’ of the actual decorator method once Heka configuration has actually beenloaded. The first time the decorated function is invoked, the predicate method will be called. If the result isTrue, then heka_call (intended to be implemented by subclasses) will be used as the decorator. If the predicatereturns False, then _invoke (which by default does nothing but call the wrapped function) will be used as thedecorator.

heka_call(*args, **kwargs)Actual heka activity happens here. Implemented by subclasses.

predicate()Predicate used to determine if function is rebound during the rebind process

True return value will rebind such that self.heka_call becomes the decorator function, False will rebindsuch that self._invoke becomes the decorator function.

set_fn(fn)Sets the function and stores the full dotted notation fn name for later use.#

Parameters fn – Actual function that we are decorating.

heka.decorators.util.return_fq_name(func, klass=None)Resolve a fully qualified name for a function or method

class heka.decorators.stats.incr_count(*args, **kwargs)Lazily decorate any callable w/ a wrapper that will increment a heka counter whenever the callable is invoked.

class heka.decorators.stats.timeit(*args, **kwargs)Lazily decorate any callable with a heka timer.

2.8 Exceptions

exception heka.exceptions.EnvironmentNotFoundError(varname)Raised when an environment variable is not found

exception heka.exceptions.MethodNotFoundError(msg=’‘)Raised when a method lookup fails

• genindex

• modindex

• search

2.7. Decorators 17

Python Module Index

hheka.client, 12heka.config, 11heka.decorators.base, 17heka.decorators.stats, 17heka.decorators.util, 17heka.exceptions, 17heka.filters, 16heka.streams.dev, 15heka.streams.tcp, 16heka.streams.udp, 15

19


20 Python Module Index

Index

Symbols__init__() (heka.streams.tcp.TcpStream method), 16__init__() (heka.streams.udp.UdpStream method), 15__weakref__ (heka.streams.dev.DebugCaptureStream at-

tribute), 15__weakref__ (heka.streams.dev.FileStream attribute), 15__weakref__ (heka.streams.dev.StdOutStream attribute),

15__weakref__ (heka.streams.tcp.TcpStream attribute), 16__weakref__ (heka.streams.udp.UdpStream attribute), 15_convert() (in module heka.config), 11

Aadd_method() (heka.client.HekaClient method), 12ALERT (heka.client.SEVERITY attribute), 14

Cclient_from_dict_config() (in module heka.config), 11client_from_stream_config() (in module heka.config), 12client_from_text_config() (in module heka.config), 12CRITICAL (heka.client.SEVERITY attribute), 14critical() (heka.client.HekaClient method), 13

DDEBUG (heka.client.SEVERITY attribute), 14debug() (heka.client.HekaClient method), 13DebugCaptureStream (class in heka.streams.dev), 15dict_from_stream_config() (in module heka.config), 12

EEMERGENCY (heka.client.SEVERITY attribute), 14env_version (heka.client.HekaClient attribute), 13EnvironmentNotFoundError, 17ERROR (heka.client.SEVERITY attribute), 15error() (heka.client.HekaClient method), 13exception() (heka.client.HekaClient method), 13

FFileStream (class in heka.streams.dev), 15

Ggauge() (heka.client.HekaClient method), 13

Hheka() (heka.client.HekaClient method), 13heka.client (module), 12heka.config (module), 11heka.decorators.base (module), 17heka.decorators.stats (module), 17heka.decorators.util (module), 17heka.exceptions (module), 17heka.filters (module), 16heka.streams.dev (module), 15heka.streams.tcp (module), 16heka.streams.udp (module), 15heka_call() (heka.decorators.base.HekaDecorator

method), 17HekaClient (class in heka.client), 12HekaDecorator (class in heka.decorators.base), 17

Iincr() (heka.client.HekaClient method), 13incr_count (class in heka.decorators.stats), 17info() (heka.client.HekaClient method), 13INFORMATIONAL (heka.client.SEVERITY attribute),

15is_active (heka.client.HekaClient attribute), 14

MMethodNotFoundError, 17

Nnest_prefixes() (in module heka.config), 12NOTICE (heka.client.SEVERITY attribute), 15

Ppredicate() (heka.decorators.base.HekaDecorator

method), 17ProtobufEncoder (class in heka.encoders), 16

21


Rreturn_fq_name() (in module heka.decorators.util), 17

Ssend_message() (heka.client.HekaClient method), 14set_fn() (heka.decorators.base.HekaDecorator method),

17setup() (heka.client.HekaClient method), 14SEVERITY (class in heka.client), 14severity_max_provider() (in module heka.filters), 16StdlibPayloadEncoder (class in heka.encoders), 16StdOutStream (class in heka.streams.dev), 15

TTcpStream (class in heka.streams.tcp), 16timeit (class in heka.decorators.stats), 17timer() (heka.client.HekaClient method), 14timer_send() (heka.client.HekaClient method), 14type_blacklist_provider() (in module heka.filters), 16type_severity_max_provider() (in module heka.filters),

16type_whitelist_provider() (in module heka.filters), 16

UUdpStream (class in heka.streams.udp), 15

Wwarn() (heka.client.HekaClient method), 14WARNING (heka.client.SEVERITY attribute), 15write() (heka.streams.dev.DebugCaptureStream method),

15write() (heka.streams.tcp.TcpStream method), 16write() (heka.streams.udp.UdpStream method), 15

22 Index

heka-py Documentation

Documents

Transcript of heka-py Documentation