CloudStack Metering – Working with the Usage Data

CloudStack Metering Working with the Usage Data

Tariq IqbalSenior Consultant

[email protected]: @TariqIqbal_ @ShapeBlue

@ShapeBlue #CloudStack #CCC13

CloudStack Collaboration Conference 2013

Involved with CloudStack before donation to Apache Built and deployed CloudStack/CloudPlatform based

clouds for Enterprises and Service Providers globally Specialise in integrating CloudStack with Business

Support and Operational Support systems, which include: CloudPortals (including Payment Gateways, Fraud Control

solutions) Ticketing Systems CRM/ERP Systems Monitoring Solutions

About Me



“ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform

integrator & consultancy”

About ShapeBlue



CloudStack metering - What and how? CloudStack configuration to enable Usage capture Usage Data generation process Methods of accessing the Usage Data Real World Use Cases

Overview



Organisations need to be able to meter their cloud's consumption and monetize the cloud offerings by charging for usage

Majority of organisations already have existing billing or business support systems and just need access to CloudStack Usage Data

Organisations tend to have different billing criteria and charging models requiring a comprehensive integrated solution

Metering Requirements



CloudStack Usage?

CloudStack Metering



CloudStack does not offer any integrated usage monitoring or billing/chargeback capability

Root Admin can view System Capacity

Domain Admin and Users can view VM status

Metering in CloudStack today



CloudStack includes a Usage Server that creates summary usage records for the various resources consumed in CloudStack

The Usage Server is an optional, separately-installed component called ‘cloudstack-usage’

The Usage Server runs at least once per day. It can be configured to run multiple times per day and it’s behaviour is controlled by the following global configuration settings:

•

Cloud Usage Server



As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in the ‘cloud’ database table: usage_events

There are 110 different Events: (51 shown)

CloudStack Events



The Usage Server schedules a job at the time specified by usage.stats.job.exec.time parameter. This job queries the ‘cloud’ database table: usage_events and generates summary usage records in the ‘cloud_usage’ database table: cloud_usage as follows:1. CloudStack events have been logged into the cloud.usage_event table2. cloudstack_usage job gets a list of the latest usage events3. Inserts the latest usage events into the cloud_usage.usage_events table4. Helper tables populated from cloud_usage.usage_event table after

parsing5. cloud_usage.cloud_usage table populated with aggregation range wise

data. Helper table data is used here

Usage Record Generation



The Usage Server collects 14 different types of resource usage:

Usage Types



The Usage Records show the amount of resources consumed by guest instances. The Usage Record format for Allocated & Running VMs is shown below: The other 7 Usage Record formats are:

Network Usage IP Address Disk Volume Template, ISO & Snapshot Load Balancer & Port Fwd Network Offering VPN User

Usage Records



Usage Record Example



Accessing Usage Data

CloudStack Metering



The usage records can be accessed through the CloudStack API

This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed

CloudStack provides the following Usage API calls with Root Admin access and supports XML and JSON response formats listUsageTypes – list of Usage Types accessible generateUsageRecords - asynchronous usage record

generation listUsageRecords - provides Usage records for a date

range

CloudStack API



The base command is:http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=yyyy-MM-dd&enddate=yyyy-MM-dd&signature

Append the various options:

ListUsageRecords API



<listusagerecordsresponse> <count>1</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord>

</listusagerecordsresponse>

ListUsageRecords Response



CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API

It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing

Filtering can be used to limit the result set It supports argument passing and shell automation The host, port, apikey and secretkey can be

configured on setup

CloudMonkey CLI



CloudMonkey list usagerecordsThe command to retrieve Usage Records is:

cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f-080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2



The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack

Usage Data can be exported in a CSV format from a SELECT query

The SQL query can become complex if the ID fields in the data need to be de-referenced

Typically multiple passes of the query results may be required to extract the required information

SQL



Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system.

Considerations: Do you actually need all the Usage Data for all Resources? Any UUID values of the CloudStack resources should also

be imported into the target system for reference purposes Decide which system will be the master for the user

account data and if synchronisation is required

Mediation



Real World Use Cases

CloudStack Metering



MS Excel can be used to analyse CloudStack Usage Records The usage records are imported from the cloud_usage.

cloud_usage table into MS Excel via an MySQL ODBC connection

Either PivotTables or reports can be used to present the data.

This is convenient and great for integration testing between CloudStackand external systems.

MS Excel



Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform.

The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM.

Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the tenants billing cycle.

Citrix CloudPortal



splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs

We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data

Setup feature-rich dashboards and the capability to drill down intothe Usage Data for BI reporting

splunk>



CloudStack Developers Guide, API Documentation & Wiki: http://cloudstack.apache.org/

CloudMonkey https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudm

onkey-cli.html How to Use MS Excel to Analyze CloudStack Usage

Records: http://support.citrix.com/article/CTX132030

CloudStack Mailing Lists (Users, Development, Marketing): http://cloudstack.apache.org/mailing-lists.html

Further Information

http://cloudstack.apache.org/

http://cloudstack.apache.org/

https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudmonkey-cli.html

https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudmonkey-cli.html

http://support.citrix.com/article/CTX132030

http://support.citrix.com/article/CTX132030

http://cloudstack.apache.org/mailing-lists.html

CloudStack Metering Working with the Usage Data

Tariq IqbalSenior Consultant

[email protected]: @TariqIqbal_ @ShapeBlue

CloudStack Metering – Working with the Usage Data

Technology

Transcript of CloudStack Metering – Working with the Usage Data