Spotfire Integration & Dynamic Output creation

Dynamic Creation and/or Content Modification of Spotfire output

(DXP files)

Proposed Integration Solution with Pipeline Pilot

Dynamic Creation and/or Content Modification of Spotfire output

(DXP files)

Proposed Integration Solution with Pipeline Pilot

© 2009 Accelrys, Inc.

What are Spotfire Output (or DXP) files?What are Spotfire Output (or DXP) files?

• Software and Use

o TIBCO Spotfire is a comprehensive software platform that allows customers to access, analyze and create dynamic report(s) on their data.

o A report that is created/opened using Spotfire is referred to as an Analysis Document and when saved is given an extension of “.DXP”.

o This DXP file not only contains a series of metadata information, but it also contains references to the data itself and to various other components being part of the document, such as pages, filtering, bookmarks etc.

2

© 2009 Accelrys, Inc. 3

TIBCO Spotfire ArchitectureTIBCO Spotfire Architecture

TIBCO Spotfire Automation Services

Spotfire Automation Services provides a platform and tools for automating TIBCO Spotfire without user interaction and visible user interface.


TIBCO Spotfire Developer (SDK)TIBCO Spotfire Developer (SDK)

TIBCO Spotfire Developer has a complete set of application programming interfaces (APIs) for not only configuring and automating the platform, but extending it with entirely new capabilities as well.

The SDK provides required Spotfire developer resources: the Spotfire Extension Project Template, development assemblies, example projects, and the Package Builder application wrapping the extensions for deployment

A Spotfire extension is the smallest functional unit added to the platform. It is developed in Visual Studio® and is included in a Spotfire add-in, enabling versioning, licensing, deployment and loading.


TIBCO Spotfire APIs & Integration with PPTIBCO Spotfire APIs & Integration with PP

(PP Custom Extension)

Pipeline Pilot


Automation Services - TerminologyAutomation Services - Terminology

Task An Automation Services Task is the building block for creating a complex Job. Automation Services comes with a number of predefined tasks such as Apply Bookmark or Send Email, but since the platform is extendible, additional tasks may be supplied by other vendors or by end user’s IT organization.

Job An Automation Services Job is a collection of Tasks to be executed in a sequential manner. A Job is stored in a Job File.

Job File A Job File is a human-readable XML defining a Job.

Job Builder The Automation Services Job Builder is an application that runs within Spotfire Professional and provides an easy-to-use interface for creating and editing Jobs


Spotfire Automation Services - Architecture Spotfire Automation Services - Architecture

The Job is normally created in the Job Builder and saved as an XML Job File.

This file is then sent to the Automation Services Web Service for execution- from the Job Builder window (for testing),- or using the provided ClientJobSender tool, - or by your own custom tool.

For each Job in XML format sent to the Automation Services Web Service for execution, a Job Executor process is started.

This internal Job Executor logs on to a Spotfire Analytics Server using credentials stored in the Spotfire.Dxp.Automation.Launcher.exe.config file, executes the tasks one by one and then exits when a task fails or all tasks have completed successfully

The PP Custom Task will be used when creating a Job and saving the Job file. The ClientJobSender tool will be used by Pipeline Pilot to send the Job file to the Automation Services Web Service.

(Pipeline Pilot)

( PP Custom Ext. )


Automation Services – Scheduling a JobAutomation Services – Scheduling a Job

To programmatically activate a Job file, Spotfire has provided an application that excepts two arguments. The first parameter is the URL to the web server where the Automation Services is installed and the second argument is the path to the Job file, consisting of the custom task(s).

(Spotfire.Dxp.Automation.ClientJobSender.exe)

For example:

First Argument: http://localhost/SpotfireAutomation/JobExecutor.asmx

Second Argument: pathToJobFile/jobfilename.xml

"c:\temp\dxpdata\Spotfire.Dxp.Automation.ClientJobSender.exe" "http://laptopdayton/SpotfireAutomation/JobExecutor.asmx" "c:\temp\dxpdata\DXPRequestJob.xml"


Solution – Develop Communication MessageSolution – Develop Communication Message

Spotfire Automation

Services

PP Custom TaskCapable of creating a

new DXP file or modifying an exiting

DXP by using the Spotfire APIs

Pipeline Pilot

ProtocolCapable of activating thePP Custom Task that is

part of the Spotfire Automation Services.

Using Spotfire provided ClientJobSender tool.

1. An XML formatted message will be defined to enable Pipeline Pilot to specify details for the type of DXP file to generate or the updates to make to an existing DXP file.

2. The PP Custom Task will also be aware of this custom developed XML formatted message and will be translate them into APIs calls to create the DXP file dynamically.


Solution – Transmitting / Receiving MessagesSolution – Transmitting / Receiving Messages

Spotfire Automati

on Services

PP Custom Task

Pipeline Pilot

Protocol

Activate the PP Custom Task job using Spotfire provided

ClientJobSender tool.

• Pipeline Pilot formats and places a XML formatted message requesting a DXP file. The details for the type of DXP file to create, the data file to use and other required information is stored in this file.

• Pipeline Pilot activates the PP Custom Task to process it’s request for a DXP file.

• The PP Custom Task retrieves the DXPRequest.xml file, interprets the instructions from Pile Pilot and using the APIs create/modify a DXP file.

• The resultant DXP is stored in the Results folder.

• Pipeline Pilot retrieves the resulting DXP file.

REQUEST FOLDERAccessible to both Spotfire and Pipeline

Pilot.DXPRequest.xml

1

3

1

2

3

2

RESULTS FOLDERAccessible to both Spotfire and Pipeline

Pilot.filename.dxp

5

4

4 5


Implementing the Solution – DXP Request Message formatImplementing the Solution – DXP Request Message format

1. DXP Request Message: An XML formatted message structure will be defined for

Pipeline Pilot to request customized DXP files from the Custom Spotfire task.

Both Pipeline Pilot and the Custom Spotfire Task will need to be aware of the message structure.

2. Communication Medium: Some form of communication mechanism is required

between Pipeline Pilot and the Spotfire Custom Task. One option is to use a common folder or folders to store the requests and the results. Pipeline Pilot and the Spotfire App would each monitor these folder(s) periodically or when activated.


Implementing the Solution – Spotfire RelatedImplementing the Solution – Spotfire Related

1. Extension: A custom extension will be developed as a Spotfire Add-in using the Spotfire SDK.

2. Task: The custom Spotfire Add-in is then registered as an Automation Services Task.

3. Job Builder : The Automation Services Job Builder will then be used for defining a Job consisting of this custom task and a Job File will be saved.


Implementing a Solution: Pipeline Pilot RelatedImplementing a Solution: Pipeline Pilot Related

Pipeline Pilot Components:

1.Components will be developed to support this functionality.

2.As a minimum, a component that can create a Spotfire Analysis Request file using the predefined message structure.

3.Another component to move the Analysis request file to the predefine Spotfire folder and activate the Automation Services Job.

4.And yet another component that will retrieve the resulting analysis document (.DXP file) that the custom task has created.


Proof-Of-Concept DemoProof-Of-Concept Demo

Communication Between Spotfire and Pipeline Pilot:

Three folder were created that were accessible to both Pipeline Pilot and Spotfire.

1)Folder named DXPRequests: This folder is in which the XML formatted analysis requests are stored by Pipeline Pilot and retrieved by the custom task in Spotfire.

2)Folder named DXPResults: This folder is in which the Spotfire generated analysis document (.DXP file) is stored and retrieved by Pipeline Pilot.

3)Folder Named DXPData: This is a folder is used for storing sample data files and preconfigured XML formatted requests (that will be used in the demo.)



Spotfire Setup:

1)The custom task was registered as an Spotfire Automation Services task.

2)Using the Job Builder, a Job file was created that consisted on our custom Automation Services task.

3)The resulting XML formatted Job File was saved in the DXPData folder. (DXPRequestJob.xml)

4)A copy of the Spotfire Spotfire.Dxp.Automation.ClientJobSender.exe was saved in the DXPData folder.



Pipeline Pilot Setup:

1)Two sample Analysis Request files were created and stored in the DXPdata folder. The contents of one of the files is shown below. (This request file is for the demo only and the format of the final message is still to be developed.)

<?xml version="1.0" encoding="utf-8"?><PPDXPRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-

instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <dxpRequest>2</dxpRequest> <xDataColumnName>Cost</xDataColumnName> <yDataColumnName>Sales</yDataColumnName>

<plotDataFile>c:\temp\dxpdata\SalesData.txt</plotDataFile> <dxpResultFile>c:\temp\dxpresults\SalesDataScatterPlot.dxp</dxpResultFile>

</PPDXPRequest>

•The custom task is hard coded to recognize that dxpRequest code of 2 is for a scatter plot and that the data source is the file specified in plotDataFile node and that the resultant DXP file is to be stored as specified in the dxpResultFile node.



(Cont.) Pipeline Pilot Setup:

Three components were created to

1)execute a command line to copy a predefined request file from the DXPData folder to the DXPRequestFile folder.

2)to execute a command line to activate the Automation Services Job containing our custom task.

3)to open the resulting analysis (.DXP) file stored in the DXPResults folder by Spotfire.



Pipeline Pilot Setup:

1)Using these simple components the following protocol was created.



Execution Sequence: 1) Create and Submit Analysis Request (file)

On activating the protocol, a user is requested to selected one of the two predefine analysis request files. The selected request file is copied to the DXPData folder and renamed to dxprequest.xml – This is the name of a file that the custom task on Spotfire will look for.



Execution Sequence: 2) Activate Automation Services Job

Once the request file is copied to the DXPRequests folder, a another component activates the Automation Services Job on Spotfire to process this request. (there are other options for activating the Automation Services Job besides the method used in this demo.)

The command line executed by this component is

"c:\temp\dxpdata\Spotfire.Dxp.Automation.ClientJobSender.exe" "http://laptopdayton/SpotfireAutomation/JobExecutor.asmx" "c:\temp\dxpdata\DXPRequestJob.xml“



Execution Sequence: 3) Execute Analysis Instructions (Spotfire)

The XML formatted instructions from the file and the Spotfire APIs are used for modifying an existing analysis document or for creating a new analysis document.

In our example the contents of the XML Request file is as follows

<?xml version="1.0" encoding="utf-8"?><PPDXPRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <dxpRequest>2</dxpRequest> A Request Type of 2 was implied to mean a Scatter Plot <xDataColumnName>Cost</xDataColumnName> Column name of the x data values

<yDataColumnName>Sales</yDataColumnName> Column name for the y data values

<plotDataFile>c:\temp\dxpdata\SalesData.txt</plotDataFile> Location of the source data

<dxpResultFile>c:\temp\dxpresults\SalesDataScatterPlot.dxp</dxpResultFile> location for the resulting analysis document.

</PPDXPRequest>



Cont. When the Automation Services Job is activated, it executes the code of the custom task and also provides access to the running TIBCO Spotfire instance.

A document opened in a running instance of TIBCO Spotfire is referred to as an Analysis Document.

The document not only contains a series of metadata information, but it also contains references to the data itself, and to various other components being part of the document, such as pages, filtering, bookmarks etc.

Since a data file was specified, the custom task uses the Open method of the Analysis Application and the specified data file.

We now have a default analysis document that can be customized with the requested visualization.

In this request file the requested visualization is for a scatter plot and so the available method(s) are used for adding a scatter plot visualization to the analysis document using the data file and the X and Y data column names.

The analysis document is then saved using the name/location specified in the request file.



Execution Sequence: 4) Receive Requested Analysis (.DXP File)

Once control is returned, the resulting analysis file will have been stored in the DXPResults folder by the custom task that was executed by the Spotfire Automations Services Job.



Execution Sequence: 5) View Analysis Document (.DXP file)And finally the analysis file is opened by a component executing the following command line

c:\temp\dxpresults\salesdatascatterplot.dxp

Note: The following plot was created using the default template provided by Spotfire’s API. The Filters; The Page and the Details-on-Demand were preconfigured but could be customized. In this case the data file and name of the X and Y columns were provided.


Phase-II Phase-II

1. Gather a list of all possible DXP file requests that are required from Pipeline Pilot.

2. Using this data, an XML formatted message structure will need to be to determined that would be able to support all the requests.

3. Determine if these requests can be implemented using Spotfire APIs.

4. Develop a library of Pipeline Pilot components that an end user could use with or without modifications.

5. Develop the custom task to interpret these DXP requests.

Spotfire Integration & Dynamic Output creation

Technology

Transcript of Spotfire Integration & Dynamic Output creation