Whitepaper A Developers Guide to Accelerating Microsoft Excel Models With Platform Symphony

8/3/2019 Whitepaper A Developers Guide to Accelerating Microsoft Excel Models With Platform Symphony

1/16

A Developers Guide to Accelerating

Microsoft Excel models with

Platform Symphony

PLATFORM COMPUTING WHITEPAPER - APRIL 2011

Gord SissonsPlatform [email protected]


2/16

4/6/11 1

Contents

1. Introduction ......................................................................................... 22. Overview ............................................................................................ 33. Common Excel Integration Approaches ......................................................... 4

Pattern #1: Invoking a Custom Service ......................................................... 4Pattern #2: Scripts or Executables as Tasks ................................................... 7Pattern #3: Excel Instances on the Grid ....................................................... 9

Pattern #4: Extending Excel Services UDFs ................................................... 10Pattern #5: Hybrid Approaches ................................................................. 13

4. Summary .............................................................................................. 14


3/16

4/6/11 2

1. Introduction

This white paper illustrates how Platform Symphony can help developers improve theperformance of Microsoft Excel based computations.

After providing some context on the business challenge, this paper examines severalalternative approaches to adapting Excel applications to a distributed grid-computingenvironment.

Developers may also wish to review the Platform whitepaper A Developers Guide to BuildingHigh Performance Service-Oriented Applications for a more complete discussion of PlatformSymphony, its unique architecture, and the Platform Symphony Developer Edition.


4/16

4/6/11 3

2. Overview

Microsoft Excel is widely used in a range ofbusiness applications. With a rich set of built-

in functions, a straightforward programmingmodel, and a large library of available third-party add-ins, Excel has become a staple toolfor analysts running numerical simulations inareas including risk management, actuarialanalysis and monte carlo simulation.

The accessibility and ease of use of Excel hasresulted in many financial servicesorganizations making extensive use of Excelbased models to automate complex repetitivetasks such as extracting market data from

real-time sources and computing pricingscenarios and risk positions. With increasedtrading volumes, increased electronificationof exchanges, and ever more complex financialproducts, the need to run more sophisticatedanalysis in a short time, taking into accountmore risk factors has become a critical sourceof competitive advantage. Particularly forfront-office applications where response timesare of critical importance.

With Excel 2007 and Excel Services (a

SharePoint technology allowing Excel 2007workbooks to be accessed through a browser),Microsoft have recognized these uses of Excel,and have introduced a number of powerfulfeatures intended to make it more practical todeploy Excel based worksheets as sharedcalculation services. While there is richfunctionality in .NET for developers creatingnew applications, a great number of existingmodels written in Excel 2003 and olderversions are in widespread use. These modelsoften make extensive use of VBA (Visual Basic

for Applications) to perform calculations andintegrate custom code implement as Excel LinkLibrary add-ins (XLLs).

While flexible and powerful, VBA is aninterpreted language and runs slowly its notuncommon for complex spreadsheets to run

literally for hours on a single computer to runa complex risk calculation. While it would benice to be able to simply re-write olderspreadsheet models using compiled languagesand more modern tools and approaches, itsseldom that simple. Regulatory requirementsdictate the need for calculation repeatability,meaning that firms are required to retain theirlegacy Excel models along with historicalsnapshots of data. Also, even if it werepractical to re-architect these spreadsheet-based models around a single framework, it

would be both expensive and time-consumingto do so.

Most firms employ a variety of calculators andsimulation tools both commercial and in-housedeveloped. While many run on Microsoftplatforms, others run on commercial UNIX orLinux environments, and calculation servicesare implemented in a variety of programminglanguages including Java, C, C++ and C#making flexibility critical.

Platform Symphony integrates seamlessly intoa .NET environment, but also offers a flexible,language and platform agnostic programmingmodel. This open and flexible approach makesPlatform Symphony the industrys best solutionfor clients needing practical solutions toaccelerating a variety of Excel modelsimplemented using multiple Excel versions andtechnology approaches.


5/16

4/6/11 4

3. Common Excel Integration Approaches

Table 1 shows some common design patterns used to accelerate Excel based models by runningcomputations in parallel. While there are other potential integration approaches, the remainder of

this paper focuses on explaining the patterns below and showing how a developer can easilyparallelize computations and improve performance using Platform Symphony. In some cases we haveprovided coding examples as well so that the reader can appreciate how straightforward theintegration effort is.

Table 1: common approaches to accelerating Microsoft Excel models

Pattern #1CustomDevelopedServices

Pattern #2Command lineutilities astasks

Pattern #3Excel Instanceson the Grid

Pattern #4Extending ExcelServices viaUDFs

Pattern #5HybridDeploymentScenarios

Description Excelspreadsheetcalls adistributedcomputeservices via theSymphony COMAPI

An Excel clientinvokes servicesthat are simplescripts orbinaries runningon computenodes

Multiple Excelinstances run inparallel called byclientspreadsheets orother clients

Web-basedclients accessExcel Services viasharepointemploying UDFsto distributecomputations toa grid

Combinedapproaches sharepointservices usingUDFs to invokemultiple serviceswritten in Java,C++ or running asother Excelservices

Excel Requiredon ComputeNodes

no no yes no optional

Server-sideEnvironment

Windows, UNIXor Linux

Windows, UNIXor Linux

Windows(running Excel)

Windows, UNIX orLinux Excelrunning on aSharepoint server

Windows, UNIXor Linux orcompute hostsrunning Excel

Client-sideDeveloperEnvironment

Excel Excel Excel, C, C++,VBA, VB6, Java

Excel, C, C++,VBA, VB6, Java

Excel, C, C++,VBA, VB6, Java

Developer EffortRequired

low very low medium low low


6/16

4/6/11 5

Pattern #1: Invoking a Custom Service

To avoid the relatively poor performance of VBA scripts, Excel add-ins are often written in languagessuch as C++ or C# using Visual Studio or other tools. Once these compiled add-in functions (referredto as XLLs) are associated with a spreadsheet, calculations run at native speed providing a significantperformance advantage over code written in VBA. Performance is still constrained however by theresources of a single machine, and multiple calls to these compiled XLLs will run serially within aspreadsheet. Excel users seeking greater performance will inevitably want to distribute thecalculations performed by these high-performance XLLs and run them in a service oriented modelwhere calculations can take place in parallel harnessing the power of several machines runningconcurrently. This approach is depicted in Figure 1.

Figure 1: Distributing XLL / add-in Services to a Grid

The code fragment below illustrates how VBA code in the Excel worksheet can directly open aconnection to a Platform Symphony Service, start a session and send tasks to a hypothetical PlatformSymphony managed compute service called OptionChain() running on the grid. This OptionChain()function may have been previously implemented in an XLL. Platform Symphony hides considerablecomplexity from the developer, starting the compute service on appropriate nodes based on demandand in accordance with policy, and handling error conditions that inevitably arise when running aservice in production and making them transparent to both the developer and the application user.

Open a connection to Platform SymphonyDim connection As CsoamConnectionSet connection = soamApi.Connect(OptionChain, callback) Open a session on the gridDim session As CSoamSessionSet session = connection.CreateSession(attributes) send parameter set to the compute serviceFor k = 0 To taskToSend - 1Dim message As MyMessageSet message = New MyMessage..Dim inputHandler As CSoamTaskInputHandleSet inputHandler = session.SendTaskInput(message)

Next K

Once each of the tasks have been sent to the compute service, Platform Symphony can receive theresults of these calculations using a few more lines of code.


7/16

4/6/11 6

Adapting existing code to run as a Platform Symphony service is similarly straightforward. The codeexample below shows how our sample OptionChain() service written in C++ is implemented as aPlatform Symphony Service. The C++ code is wrapped in a Platform Symphony Service Containerclass. When Platform Symphony invokes the calculation service, it will pass a taskContext to thePlatform Symphony service. The onInvoke() method retrieves the calculation parameters from

Platform Symphony, calls the existing OptionChain() code with the received parameter set, and thenreturns the results of the calculation to Platform Symphony via the SetOuputMessage() method. Thecode modifications required to grid-enable the service are shown below. Note that the main methodsimply creates an instance of the service class and invokes the run() method so that the new serviceis ready to receive work from Platform Symphony. Platform Symphony automatically handles startingand stopping services instances based on demand and sharing policies reflecting how various usersand groups are entitled to share the virtualized resources of the grid.

A Sample Compute Service Adapted to Platform Symphony

classMyServiceContainer : public ServiceContainer{

public:

virtualvoidonInvoke (TaskContextPtr& taskContext) throw(SoamException){

MyMessage inMsg, outMsg;taskContext->populateTaskInput(inMsg);..OptionChain optc(istr);optc.Value();..taskContext->setOutputMessage(outMsg);

}};

int main(int argc, char* argv[])

{ int retVal = 0;try{

// Create the container and run itMyServiceContainer myContainer;myContainer.run(argc, argv);

}catch(SoamException& exp){

// report exception to stdoutcout


8/16

4/6/11 7

Pattern #2: Scripts or Executables as Tasks

Platform Symphony execution tasks are a unique and powerful feature that allow existingexecutables to be made callable as tasks in a grid computing environment. Using this approach,organizations can realize the benefits of Platform Symphony without changing application code. Thisis particularly useful in cases where source code for a service is not available, or developers simplydo not have the time to integrate their application using the Platform Symphony API.

Workload management systems including Platform LSF and Sun Grid Engine have for a long-timesupported the distribution of command-line oriented workloads to compute nodes on a cluster.Unlike these solutions however, Platform Symphony fully abstracts these binary applications orscripts as service oriented tasks, making them callable in a variety of ways including via the clientAPI, through the Platform Symphony Management console or via a special command line utilityprovided for this purpose called symexec. Using Platform Symphony execution tasks, developers canavoid changing their server-side code; however they can still realize all of the benefits of a service-oriented application model including session management, service reliability and automated servicemanagement. The option to use the client-side symexec component to invoke the service providesdevelopers the option of avoiding software development on the client side as well. Parameters canbe sent to the execution service in a variety of ways including as scriptable name value pairsconfigured in environment variables.

Figure 2 below illustrates how this works conceptually. The Platform Symphony client opens aconnection to the grid as before, but connects to a service that is implemented as a PlatformSymphony Execution Service to run tasks.

Figure 2 : Invoking Server Side Scripts or Binaries from an Excel application


9/16

4/6/11 8

The steps involved from the client perspective are:

The client-side application opens a connection to the Platform Symphony Service via the PlatformSymphony client-side API or the provided symexec utility. A client-side Excel application canavoid the use of the Platform Symphony API by using VBA functions to invoke symexec with values

extracted from spreadsheet cells and constructed dynamically at run-time symexec can becalled with arguments including create, open, send, fetch, close and run so that executiontasks can be run on the service side without needing to use the client-side API at all.

For example in VB6:

Call Shell cmd.exe \\server\path_to_symphony\symexec send + args

Or in VB.NET

System.Diagnostics.Process.Start(cmd.exe

\\server\path_to_symphony\symexec send +args)

The execution task can have an optional pre-execution command configured via the applicationprofile useful for setting up the environment prior to running the service.

The client will then send tasks to the execution service including an executable string witharguments or an optional execution task context including environment variables set as namevalue pairs.

Upon receiving input from the client, the Platform Symphony execution service will spawn theexecution task as a process and execute it with the input arguments provided.

When the execution task has completed, the exit code of the process and task output is sent backto Platform Symphony and an optional post-execution script may be run. Error handling is

configurable in the XML-based application profile.

Platform Symphony execution tasks are a unique and powerful Platform Symphony capability thatprovide developers the freedom to make virtually any application callable on the service side.Moreover, while weve described the case of calling service side tasks from an Excel client above, inpractice these tasks can be invoked from the client in a variety of ways including through scripts onWindows or Linux or via .NET applications.


10/16

4/6/11 9

Pattern #3: Excel Instances on the Grid

While Excel Services and Excel 2007 provide exciting new functionality allowing Excel to be run as aservice, many organizations have significant investments in complex Excel worksheets written inolder environments including Excel 2000, Excel 2003 etc. These legacy spreadsheets often makeextensive use of VBA, database access methods and in-house or commercially developed add-incomponents. While there is a need to make these existing models run more quickly, organizationssimply do not have the time or resources to re-write and re-validate all of these models.

To address the need to grid enable complex spreadsheet models making extensive use of VBA,Platform offers a grid integration option referred to as the Platform Symphony Connector forMicrosoft Excel. This integration option allows Excel 2000, 2002 and 2003 worksheets to run asencapsulated services in Platform Symphony. This means that calculations can be performed inparallel on compute hosts on a cluster without the need to modify the spreadsheets themselves inmost cases. In this grid deployment model, Excel action runs as a managed Platform Symphonyservice on the cluster.

The Platform Symphony Connector for Microsoft Excel includes a Platform Symphony service that actsas a wrapper for Microsoft Excel, and allows information to be passed from a Platform Symphonyclient including the spreadsheet name, a macro name, and execution related parameters for themacro. Using this approach, spreadsheets may be implemented as services and called by a client-sideinstance of Microsoft Excel acting as a control sheet calling server-side macros with subsets of thedata that needs to be computed. Because the spreadsheet models can work on subsets of the data inparallel, the time to run these models is dramatically accelerated.

Figure 3 : Wrapping Worksheets as callable Platform Symphony Services

The Platform Symphony Connector for Microsoft Excel includes numerous coding examples showinghow services implemented as spreadsheets can be made callable from client-side applications. Whilethe client-side application is frequently a spreadsheet, it could just as easily written in C++.

Platform has gone to significant effort to address the practical issues associated which deployingExcel instances as encapsulated services. A feature-rich dialog sniffer makes it straightforward to


11/16

4/6/11 10

debug issues that can impact Excel instances such as popup-dialogs warning of particular errorconditions (out of memory errors or security related settings with macros as example) that leftunaddressed would simply cause service instances to freeze and become non-responsive.

While newly developed Excel-based models may use Excel Services approaches, the PlatformSymphony Connector for Excel with its ability to run existing spreadsheets as services without

modification, is a critical part of a developers portfolio of solutions to grid-enable Excel-basedmodels.

Pattern #4: Extending Excel Services UDFs

As described earlier, Microsoft has implemented exciting new features in Excel 2007 that make iteasy to deploy Excel-based applications to the web. Based on Microsoft Office SharePoint Server2007, the Excel Services deployment model offers a number of significant advantages. Among themare:

Users are no longer required to have access to Microsoft Excel or particular worksheets to runExcel-based models

Worksheets are made inherently multi-user when deployed via SharePoint Issues such as version control are simplified since spreadsheet developers can control how

spreadsheets are presented and avoid the problem of multiple versions of spreadsheets floatingaround an enterprise.

Figure 4 depicts the SharePoint-based Excel Services architecture. Clients may interact with Exceleither via a web-browser interface or clients may interact programmatically via an exposed web-services interface. The actual Excel calculation services run inside an application server containerand draws content from the SharePoint content database as well as other external sources. In the

Excel Services architecture, user-defined functions (UDFs) can be written to extend spreadsheetfunctionality and run in the context of the Excel Services Application Container.

Figure 4 : Integrating Platform Symphony in an Excel Service Model

The use of user-defined functions has several advantages. Excel functionality can be extended via alibrary of user-defined functions that can be stored in the SharePoint database (referred to asmanaged functions) so that they are callable by multiple Excel spreadsheets. Also, user-defined


12/16

4/6/11 11

functions are very straightforward to call from within Excel worksheets and can be called in exactlythe same manner as other functions e.g. =MyFunction($A$1*3.6)

Microsoft Visual Studio includes a class library template that makes it easy to build your own user-defined functions. Manage-code UDFs require that a dynamic link library (DLL) namedMicrosoft.Office.Excel.Server.Udf.dll is present. UDFs not having this attribute are ignored by the

Excel Calculation Service.

Sample .NET C# code below illustrates how a user-defined function called MyFunction() can bewritten that calls a Platform Symphony service deployed to the grid. The Platform Symphony API for.NET is in the name space Platform.Symphony.Soam as shown in the example below. This examplecode contains the [UdfClass] and [UdfMethod] directives that make this a managed function known tothe Excel calculation service. Although some details of the code have been omitted below, it isstraightforward to write a UDF using the Platform Symphony .NET client API that will invoke anyPlatform Symphony service. A key benefit of Platform Symphony is that the service can be runningvirtually anywhere and may be implemented using a different software framework.

using System;

using Platform.Symphony.Soam;using Microsoft.Office.Excel.Server.Udf;

namespace Platform.Symphony.Clients{

[UdfClass]class SyncClient{

[UdfMethod]

static void MyFunction(string[] args){

try{

SoamFactory.Initialize();

String applicationName = "SampleApplication";

try{

connection = SoamFactory.Connect(applicationName,securityCb);

trysession = connection.CreateSession(attributes);for (int taskCount = 0; taskCount < numTasksToSend;

taskCount++){

MyMessage inputMessage = new MyMessage(taskCount,true, senddata);

TaskInputHandle input =session.SendTaskInput(taskAttr);

}

EnumItems enumItems =session.FetchTaskOutput((ulong) numTasksToSend);

foreach(TaskOutputHandle output in enumItems){

if ( output.IsSuccessful == true ){

MyMessage outputMessage =


13/16

4/6/11 12

output.GetTaskOutput() as MyMessage;

}else{

SoamException ex = output.Exception;}

}}finally{

// Mandatory session closeif (session != null){

session.Close();

}}

}finally{

if (connection != null){

connection.Close();

}}

}catch( Exception ex ){

Console.WriteLine("Exception caught");}finally{

SoamFactory.Uninitialize();

}}

}}

Some of the key benefits of using Excel Services in conjunction with Platform Symphony are:

A single user running a compute intensive spreadsheet based model via SharePoint has thepotential to consume all of the CPU resources on a SharePoint server making it unusable toothers. By implementing UDFs that interact with the Platform Symphony client API to callPlatform Symphony Services, calculations can be moved to the grid leaving the SharePoint serverfree to support a greater number of concurrent users and applications.

Using this approach, a variety of services implementing in different languages and running ondifferent technology platforms may be abstracted to users as simple spreadsheet services. Usersmay interact with the familiar well-known spreadsheet interface, while developers and system

administrators benefit from an ability to better manage code and simplify new servicedeployment.

Finally, unlike other approaches, this is an architecture that will scale. Platform Symphonysproven ability to support large numbers or current users on multi-application grids, and its abilityto shift resources in accordance with application demand and site defined policies coupled withits ability to automatically provision, start and stop application services makes it straightforwardfor administrators to scale services in multiple dimensions i.e. more users, more departments,more compute resources and more applications.


14/16

4/6/11 13

Pattern #5: Hybrid Approaches

While not really a design pattern on its own, wed be remiss if we didnt point out that there aremultiple other ways to integrate Excel-based calculation services. These tend to be hybrids of theapproaches discussed thus far.

With its any-to-any programming model Platform Symphony provides developers with freedom ofchoice. This is of critical importance since it means that technology limitations will not get in theway of the needs of the business, and developers can often choose from among several technologysolutions and approaches for a particular problem. Clients and services may be written in multiplesoftware frameworks and may run on multiple hardware and operating system platforms. In additionto supported APIs for C++, Java & .NET languages, the COM API allows VBA or VB6 scripts to callPlatform Symphony services directly, and the Platform Symphony Execution Task facility allowsscripts and binaries to be easily wrapped as services. All of these deployment choices coupled withthe ability to wrap entire spreadsheets as managed calculation services combine to provide flexibilitythat is simply unmatched in the industry.

Figure 5 - Hybrid Implementation Approaches

Figure 5 shows a scenario where a spreadsheet model is deployed to the web using SharePoint basedExcel Calculation Services, but behind the scenes we may want to actually invoke multiple instancesof a compute service in running in parallel provided by a legacy spreadsheet application running

tightly coupled macros and VBA code implemented in Excel 2003. Similarly we may want the UDFsrunning in our SharePoint calculation service to simultaneously invoke existing C++ or Java basedservices written to run on Linux or UNIX systems. Although not shown here, these calculations in turnmay want to invoke other Platform Symphony services themselves via the client API or the symexecfacility discussed earlier. Using Excel Services, the user interface is simplified so that the end-userneed only interact with a simple spreadsheet. Platform Symphony handles all of the details ofprovisioning calculation services on appropriate nodes, providing session management and sessionrecovery capability and handling any transient errors that may occur with particular taskcalculations. The end-user sees exceptionally fast response time owing to Platform Symphonys low-latency design. With 1.6 millisecond latency, 100 synchronous transactions involving the grid can be


15/16

4/6/11 14

completed with over-head of less than two tenths of a second. This level of performance andflexibility in deployment approaches is simply not possible with other Excel friendly grid computingsolutions.

4. Summary

Weve examined a number of scenarios where Excel-based models can be accelerated with PlatformSymphony both for spreadsheets making use of use of VBA tightly coupled with Excel 2000-2003worksheets and the more contemporary Excel services models where parallelizable Excelcomputations are deployed via a web interface. Platform Symphony provides several unique featuresand capabilities making it ideally suited as a grid development and deployment platform for Excelspreadsheets and other grid friendly workloads.

The freely downloadable Developer Edition (obtained fromhttp://my.platform.com/products/platform-symphony-de) provides developers and I.T.managers with everything needed to grid-enable spreadsheet models.

Platform Symphony, with the Platform Symphony Connector for Microsoft Excel component (notincluded in the developers edition) provides support for running Excel 2000 to Excel 2003spreadsheet instances as encapsulated services with a rich set of tools to aid in debugging andmanaging encapsulated spreadsheet models

Platform Symphony makes it easy to off-load compute intensive functions from the ExcelCalculation Service running on the SharePoint Server by calling Platform Symphony services viaUDFs as shown in our example.

Platform Symphony works well with 32 and 64 bit .NET environments, but also integratesseamlessly with a variety of UNIX and Linux based platforms and in-house developed softwareclients or services written in C++, Java and other support languages.

Platform Symphony provides very low latency operation in the range of 1.6 milliseconds thismeans that Platform Symphony can scale with much higher efficiency than competing gridsolutions. Low-latency is an exceptionally important requirement as it directly impacts theperformance observed in front-office environments where traders & analysts need sub-secondresponse times.

Organizations can start with just a few compute nodes with the Platform Symphony DeveloperEdition or deploy applications at very large scale. Platform Symphony has been validated to20,000 CPUs with single applications scaling with 95% efficiency to 4,000 CPUs taking advantageof multi-core nodes seamlessly

Although not dealt with in this paper, Platform Symphony provides grid administrators with a richset of tools required to manage production grids at scale including innovations such as XML basedapplication profiles and a sophisticated consumer-lender model that allows grid-based resourcesto be shared equitably according to policy and in response to demand for resources.

For organizations running compute intensive spreadsheet based models, Platform Symphony canprovided needed flexibility and agility making it easer to respond to changing business needs. Byaccommodating a variety of different use-cases and design patterns Platform Symphony minimizesthe need to new code development reducing risk, increasing developer productivity and acceleratingyour ability to gain business advantage. Finally organizations have the strategic benefit of being on atechnology platform that will scale both in terms of the number or nodes under management and interms of management tools. By supporting all of this functionality across heterogeneous grids,enterprises enjoy maximum flexibility helping ensure that business priorities are achieved.


16/16

Platform Computing is the leader in cluster, grid and cloud management software - serving more than 2,000 of the worlds most demandorganizations for over 18 years. Our workload and resource management solutions deliver IT responsiveness and lower costs for enterpriand HPC applications. Platform has strategic relationships with Cray, DellTM, HP, IBM, Intel, Microsoft, Red Hat, and SAS.Visit www.platform.com.

World Headquarters

Platform Computing Corporation3760 14th AvenueMarkham, OntarioCanada L3R 3T7Tel: +1 905 948 8448Fax: +1 905 948 9975Toll-free Tel: 1 877 528 [email protected]

Sales - Headquarters

Toll-free Tel: 1 877 710 4477Tel: +1 905 948 8448

North America

New York: +1 646 290 5070San Jose: +1 408 392 4900

Beijing: +86 10 82276000Xian: +86 029 [email protected]: +81(0)3 6302 [email protected]: +65 6307 [email protected]

Europe

Bramley: +44 (0) 1256 883756London: +44 (0) 20 3206 1470Paris: +33 (0) 1 41 10 09 20Dsseldorf: +49 2102 61039 [email protected]

Whitepaper A Developers Guide to Accelerating Microsoft Excel Models With Platform Symphony

Documents

Transcript of Whitepaper A Developers Guide to Accelerating Microsoft Excel Models With Platform Symphony