What's New with Programmability and Scripting
Jon Peck
Technical Advisor and Principal Software Engineer
Las Vegas, November 2008 Copyright (c) SPSS Inc, 2008Copyright (c) SPSS Inc, 2008
Do you have...
• Repetitive SPSS tasks?• A need to reduce manual work?• Problems you can't solve with traditional
syntax?• A need for statistical procedures not in SPSS
Statistics?• A need for higher productivity?• A need to reduce code maintenance time?
See if programmability can help...
Copyright (c) SPSS Inc, 2008
Agenda
• Programmability introduction• Four examples
– Automating repetitive work: applySyntaxToFiles
– Integrating programs and scripting: SPSSINC MODIFY TABLES
– Adding R statistical procedures: SPSSINC QUANTILE REGRESSION
– Making your own user interface with .NETThe Statistical Explorer
Copyright (c) SPSS Inc, 2008
SPSS Statistics embeds three programming languages
• Traditional syntax and Basic scripting still present• Plug-ins let you extend capabilities using
– Python
– R
– .NET languages (Windows only)
• Free plug-in downloads or get from cd• SPSS Developer Central web site provides
articles, SPSS-written modules, plug-ins and user contributions
Copyright (c) SPSS Inc, 2008
BEGIN PROGRAM PYTHON.import spssprint "Hello, Las Vegas"END PROGRAM
• Python or R program code goes in the normal SPSS Statistics syntax window
My first program
Copyright (c) SPSS Inc, 2008
• A program in the input stream can communicate with SPSS Statistics and control it and use Python or R facilities and modules (internal mode)
spss.Submit("GET FILE='c:/data/cars.sav'.")
• A Python or .NET application can embed SPSS Statistics inside itself (external mode)– User interface does not appear
• There is a lower level C API available in an SDK
Programmability combines SPSS Statistics with Python or R
Copyright (c) SPSS Inc, 2008
Python and R are open source software
• Programmability plug-ins are an optional install– They are free– They make possible tapping the work of the
Python and R communities– Python and R have license agreements– SPSS has a Freeware license agreement
• SPSS is not the owner or licensor of the Python or R software. Any user of Python or R must agree to the terms of the license agreement located on the Python or R web site. SPSS is not making any statement about the quality of the Python or R programs. SPSS fully disclaims all liability associated with your use of the Python or R programs.
Copyright (c) SPSS Inc, 2008
Programmability increases your power, flexibility, and productivity
• Generalization
– React flexibly to metadata, results, and the environment
• Automation
– Embed program logic in jobs
• Extension
– Add or extend procedures and transformations
– Tap existing R or Python statistical modules
• Integration
– Connect SPSS inputs and outputs to other agents
• And programmability is more funCopyright (c) SPSS Inc, 2008
Version 17 has new programmability and scripting features
V 15
• Processor– SPSS syntax– Python
programs– .NET
programs
• Front End– SaxBasic
scripts– COM support
V 16 Additions
• Processor– R programs– Extension
commands– Dataset class
• Front End– Basic script
upgrade– Python
scripts
V 17 Additions
• Processor– Extension
improvements– Program and
script integration– R graphics in
Viewer
• Front End– Custom Dialog
Builder– New syntax
editorCopyright (c) SPSS Inc, 2008
• Programs run in the regular syntax stream• Users can define SPSS Statistics syntax for
program and scripts via Extension mechanism.
• Users can create dialog boxes and menus using the Custom Dialog Builder.– Not just for extensions or programs
• Python and R output appears in the Viewer– plain text– pivot tables– charts
Programmability functionality is fully integrated into SPSS Statistics
Copyright (c) SPSS Inc, 2008
• New for version 17– SETSMACRO: syntax for using Variable Sets– SPSSINC APRIORI: association rules (R)– SPSSINC HETCOR: polychoric and serial correlation (R)– SPSSINC MERGE TABLES: merge pivot tables– SPSSINC MODIFY OUTPUT: outline titling and styling– SPSSINC MODIFY TABLES: pivot table styling– SPSSINC QUANTREG: quantile regression (R)– SPSSINC RAKE: adjust weights to control totals– SPSSINC RANFOR/RANPRED: random forests (R)– SPSSINC RASCH: Rasch models (R)– SPSSINC ROBUST REGR: robust regression (R)– SPSSINC TOBIT REGR: Tobit regression (R)
• For version 16 (and 17)– Seven others
There are new extension commands available from Developer Central
Copyright (c) SPSS Inc, 2008
– Write Python or R functions to implement the functionality• Use input API's to get data to Python or R• Use output API's to create pivot tables
– For extensions,• Define the syntax in an xml file• Use tools in extension.py to receive parsed output and pass to
implementing function– R extensions can be wrapped in Python code
• Use organization or author name as first word of command
– Use the Custom Dialog Builder to create the interface• The CDB is not just for extensions
– Test!– Package and distribute– Contributions to Developer Central are welcome
• Documentation available at SPSS Developer Central
You can create and share your own additions to SPSS Statistics
Copyright (c) SPSS Inc, 2008
Expand the audience by creating SPSS Statistics syntax and dialog boxes
Copyright (c) SPSS Inc, 2008
Generalize and automate work
Example I
Copyright (c) SPSS Inc, 2008
• You have syntax files and need to process datasets not known in advance every day
• applySyntaxToFiles function applies a syntax file to each file in input specification
• Optionally saves new sav and output files• Optionally creates a log• Can be run in internal or external mode• Can be integrated into a larger process
applySyntaxToFiles
Copyright (c) SPSS Inc, 2008
• Apply standard processing to an unknown set of files• Produce processed data and reports
Automating a routine process
Copyright (c) SPSS Inc, 2008
begin program.import spss, spssaux3spssaux3.applySyntaxToFiles(inputspec="c:/temp/parts/*.sav", syntax = "c:/userconf2008lasvegas/dailychecks.sps", outputdatadir = "c:/temp/processed", outputfiledir = "c:/temp/processed", logfile ="c:/temp/processed/report.txt")end program.
• dailychecks.sps could apply data cleaning rules, modify data, and create reports
• This program could be run daily through Production Mode or PES job scheduler or used interactively
Use a program to drive processing
Copyright (c) SPSS Inc, 2008
Use integrated scripting for better table presentation
Example II
Copyright (c) SPSS Inc, 2008
• TableLooks provide static formatting for entire areas of a table– data cells– row and column layers
• You want tables with formatting beyond tableLooks• Many users copy tables to Excel and manually format
them • Basic and Python Scripting provide programmatic way
to do formatting• SPSSINC MODIFY TABLES provides syntax for
extensive formatting– Eliminates need to know scripting– Uses Extension mechanism for programs and Python
scripting
SPSSINC MODIFY TABLES extension command
Copyright (c) SPSS Inc, 2008
SPSSINC MODIFY TABLES SUBTYPE='Crosstabulation' DIMENSION=ROWS SELECT='Std. Residual'/STYLES TEXTSTYLE=BOLD BACKGROUNDCOLOR=255 0 0APPLYTO='abs(x) >2'.
Use dynamic highlighting to make crosstab table easier to read
Copyright (c) SPSS Inc, 2008
• Dialog created withCustom Dialog Builder
• Generates extension command syntax
• Easy to distribute
Custom dialog boxes are easy to create
Copyright (c) SPSS Inc, 2008
SPSSINC MODIFY TABLES subtype='variables in the equation' SELECT="B" "Sig."/STYLES TEXTCOLOR = 0 0 255 BACKGROUNDCOLOR=0 255 0.
Copyright (c) SPSS Inc, 2008
Use static formatting to call out parts of a table
SPSSINC MODIFY TABLES SUBTYPE="Custom Table" SELECT = "Total" DIMENSION=ROWS/STYLES BACKGROUNDCOLOR=255 255 88 TEXTSTYLE = BOLD
Format CTABLES totals to call them out
Copyright (c) SPSS Inc, 2008
SPSSINC MODIFY TABLES SUBTYPE='Report' SELECT="<<ALL>>"/STYLES APPLYTO=DATACELLS TEXTCOLOR=255 255 255 TEXTSTYLE=BOLDCUSTOMFUNCTION="customstylefunctions.washColumnsBlue".
def washColumnsBlue(obj, i, j, numrows, numcols, section, more): mincolor=150. maxcolor=255. increment = (maxcolor - mincolor)/(numcols-1) colorvalue = round(mincolor + increment * j) obj.SetBackgroundColorAt(i,j, RGB((mincolor, mincolor, colorvalue)))
Use custom functions for special effects
Copyright (c) SPSS Inc, 2008
Copyright (c) SPSS Inc, 2008
It is possible to get carried away with this
Add R procedures seamlessly to SPSS Statistics
Example III
Copyright (c) SPSS Inc, 2008
• R – has many advanced statistical procedures– is a complex language for programming statistics– is difficult to learn
• The R plug-in makes it easy to use R packages– SPSS Statistics datasets and Viewer output can be
processed by R using plug-in– Graphical, text, and table output appear in the Viewer– New SPSS datasets can be created from R– R can communicate with SPSS via 30 API's– Integration requires writing some R wrapper code
• Plug-in is downloadable from Developer Central
R procedures can be accessed from SPSS Statistics using the R plug-in
Copyright (c) SPSS Inc, 2008
• Ordinary regression models conditional mean• Median regression is 50th quantile• Estimating quantiles is useful with varying
spread, asymmetries, outliers• Areas of application include
– empirical finance• value at risk• mutual fund investment styles• credit scoring
– school quality– demand analysis– others
Quantile regression models conditional quantiles
Copyright (c) SPSS Inc, 2008
SPSSINC QUANTILE REGRESSION extension embeds R quantreg
Copyright (c) SPSS Inc, 2008
Plots and pivot tables appear in the Viewer
Copyright (c) SPSS Inc, 2008
New datasets appear in Data Editor windows
Copyright (c) SPSS Inc, 2008
The .NET plug-in allows an easy .NET way to do your own user interface for SPSS Statistics
Example IV
Copyright (c) SPSS Inc, 2008
The Statistical Explorer uses the .NET plug-in
Copyright (c) SPSS Inc, 2008
The Statistical Explorer
Copyright (c) SPSS Inc, 2008
Statistics and chart shown depends on variable measurement level
Copyright (c) SPSS Inc, 2008
• SPSS Statistics 17 completes programmability and scripting building blocks
• Unification of programs and scripts
• Custom Dialog Builder
• Extension improvements and new extensions
• R integration and graphics in Viewer
• SPSS Developer Central is your friend
Where we have been today
Copyright (c) SPSS Inc, 2008
? ?
Questions
Copyright (c) SPSS Inc, 2008
• Generalization and automation
– applySyntaxToFiles
– SPSS MODIFY TABLES
• Extension
– SPSSINC MODIFY TABLES using integrated scripting
– SPSS QUANTREG using R
– Many new extension commands available
• Integration
– applySyntaxToFiles as part of a process
• And it's still more fun
Programmability increases your power, flexibility, and productivity
Copyright (c) SPSS Inc, 2008
Contact
Copyright (c) SPSS Inc, 2008
Top Related