© 2 0 0 9 b y H a r v e y E. H a h n S ome S ecrets of S uccessful S cripting and S tatistics h...

© 2 0 0 9 b y H a r v e y E . H a h n

S ome S ecrets of S uccessful S cripting and S tatistics h arvey h ahn

S ome S ecrets of S uccessful S cripting and S tatistics h arvey h ahn

Arlington Heights Memorial Library

Arlington Heights, Illinois

PresenterPresenter

Harvey E. Hahn

Former Manager,

Technical Services,

Arlington Heights

Memorial LibraryArlington Heights, Illinois

Program overviewProgram overview

• Part 1• Tips gained from 4 years and over 25,000

lines of AutoIt code (as well as 12 years of writing 152 OCLC macros, including III telnet applications)

• Part 2• Brief “whet your appetite” introduction to

the J programming language for processing exported Millennium data

Some

Scripting

“Secrets”

PurposePurpose

• To share helpful patterns, tips, and techniques that you can use in your scripts to automate Millennium

What is a script?What is a script?

• One or more groups of instructions

that perform a sequence of window actions,

often entering, modifying, and/or deleting data,

imitating what a person would perform

What is “dumb” scripting?What is “dumb” scripting?

• Sends text, keystrokes, and mouse clicks to one or more windows “blindly”

• Does not (and cannot) “react” to what is happening on the screen

• Millennium keystroke “macros” are simple examples of this

What is “smart” scripting?What is “smart” scripting?

• “Aware” of which window is currently displayed

• Can “react” appropriately to screen behavior

• Sends text, keystrokes, and mouse clicks to one or more windows “intelligently”

• Can “read” textual data from the screen and act upon it in some way

Why use scripts?Why use scripts?

• Computer can do what it’s good at• Speed• Repetition

• People can do what they’re good at• Judgment / Decision making

Result :Result :• Increased efficiency• Increased productivity

Script language comparisonScript language comparison

• Cost• AutoIt, AutoHotKey, VBScript, and Expect are

free• OCLC Macro Language is freely available to

OCLC customers only• Macro Express costs $40 per computer

• GUI capabilities• AutoIt, AutoHotKey, and Macro Express do

windows• VBScript, Expect , and OML do not

Other helpful freeware toolsOther helpful freeware tools

• FRHED (FRee Hex EDitor)http://www.kibria.de/frhed.html

• IrfanView (“EAR-fun-view”) (image viewer)http://www.irfanview.com/

• Any text editor (Notepad, Notepad+, NoteTab,

Win32Pad, etc.)

FRHEDFRHED

IrfanViewIrfanView

Successful scripting “secrets”Successful scripting “secrets”

• Typing keystrokes

• “Where am I?”

• Window coordinates

• Clicking buttons / tabs / icons• WYSIWYG—not! (getting window data)

Typing keystrokesTyping keystrokes

• Known issue with Microsoft Windows:• A script can send keystrokes faster than

Windows can process them• Programmers need to insert delays after

Send commands to permit Windows to catch up

• The delay value should be at least 2/10ths of a second, possibly more


• For a Send command to work successfully, a Sleep command needs to be paired with it:

Send( “John{ENTER}” )Sleep( 200 )Send( “Smith{ENTER}” )Sleep( 200 )


• More efficient approach:

Func Type( $Text )

Send( $Text )

Sleep(400) ; add pause after sending text

EndFunc

. . .

Type( “John{ENTER}” )

Type( “Smith{ENTER}” )


• Millennium scripting tip:

Always first try using a keyboard shortcut (before trying mouse clicks) because it’s the easiest and most reliable technique


• Millennium keyboard equivalents:

#100743 – function keys

#100742 – general shortcut keys

#100535-100542 – window controls and associated shortcut

keys

Typing keystrokes Millennium keyboard equivalentsTyping keystrokes Millennium keyboard equivalents

“Where am I?”“Where am I?”

• Getting the title of the active window

Why?

Identify various windows (error,processing, get info from user,

etc.)


• Waiting for a window to appear

Why?• Script must wait until a given window is

“ready for action”• Script might need to check for possible

appearance of an “optional” window• This is one of the most important aspects

of GUI scripting!



Tip:• Watch for possible “surprise” appearance

of an error message window• These are undocumented!



How?• Keep checking to see when the title of the

window has changed (but sometimes multiple windows have the same title!), or

• Wait until the window is “active” (focus) (most common approach)


• Waiting for a window (or part of a window) to change

Why?• Script must wait until a given window is “ready for

action”• Script must wait until part of a given window has

changed and is “ready for action”• This is one of the most important aspects of GUI

scripting!


• Waiting for a window (or part of a window) to change

How? (in AutoIt)

• (1) Determine a rectangular area to check• (2) Take a “snapshot” of the area with

PixelCheckSum• (3) Keep taking “snapshots” until the pixel

checksum for the area changes (compared with the original value)


• Getting pixel colors of text, background, or images (“worst case” scenario)

AutoIt syntax:$hexcolor = Hex( PixelGetColor( $x, $y ), 6 )

0 = black

FFFFFF = white

RRGGBB


• Distinguish windows with same titles

“C” “D”


• Using patterns of pixels to identify characters or positions on the screen is why it is so important to maximize windows and thus stabilize coordinates

• Coordinates being “off” by 1 or 2 pixels can make all the difference in the world!

Window coordinatesWindow coordinates

• The screen is the entire desktop

• A window is placed somewhere on the screen

• A Millennium window usually covers the entire screen


• Coordinates are zero-based from the upper left corner of a window or screen

• Coordinates inside of a window stay the same, regardless of where a window is located on the screen


98

79

79

98


• Millennium scripting tip:A window must be maximized for consistent coordinates

Example of AutoIt syntax:WinSetState( $WinName, "", @SW_MAXIMIZE )

NOTE:This command will even maximize windows which have no maximize button!



The screen resolution affects coordinates!

Always indicate your screen resolution in the script preliminaries—this helps others when using or adapting your script

Clicking buttons/tabs/iconsClicking buttons/tabs/icons

• “Wrapper” function approach:

Func ClickAt( $x, $y )

MouseClick( "left", $x, $y, 1, 0 )

Sleep(400) ; add pause after clicking mouse

EndFunc

. . .

ClickAt( 400, 300 )


• Clicking depends on the desired coordinates of the mouse pointer on a screen or window

• Coordinates may be relative to:• a given window• the full screen


Window-based coordinates


Screen-based coordinates


• To keep coordinates inside of a window stable, they should always be relative to that window (not to the screen)


• To resolve any screen/window conflicts, offsets are needed from the screen edges to a window’s upper left corner (“origin”)


X offset

Y offset


• Match coordinate references to the type of justification being used

• Add offsets to left edge for left-justified text, tabs, icons, etc.

• Subtract offsets from right edge for right-justified text, tabs, icons, etc.

WYSIWYG—not! (getting window data)WYSIWYG—not! (getting window data)

• “Reading” data off the screen is often the most difficult and challenging task when writing scripts to automate Millennium


• Select (highlight) specific text

How?• Use the standard Windows “select all” command

(Ctrl-a)• Depending on the screen context, this may select

the contents of a single row or the contents of an entire bib, item, order, or patron record


• Use the clipboard

AutoIt syntax:

$text = ClipGet() ; get data from clipboard

ClipPut( $text ) ; put data onto clipboard

Note:It is standard practice to clear the clipboard before copying any data to it


Example

Func GetHighlightedData() ClipPut( “” ) ; always clear the clipboard first! TypeCtl ( “c” ) ; copy highlighted data to the clipboard TypeCtl ( “c” ) ; duplicate needed for reliability (why??) $HighlightedData = ClipGet() ; put data into a variable Return $HighlightedData

EndFunc. . .TypeCtl( “a” ) ; select (highlight) all editable text$text = GetHighlightedData()


• Tables

Algorithm• Choose a row, using Home, End, Up, or Down or

Rapidly type a row number or

Click a row in the table • Copy the auto-highlighted data into a string• Parse the string, looking for specific data


check for this text



• To find a highlighted row in a table, don’t loop through every pixel in a column, looking for the color

• Note: Cell data in tables is 16 pixels apart. For speed purposes, check just every 8th (or even 16th) pixel in a blank area of a given data column


• Dropdown listsElements in these lists cannot be “read” by scripts

• Identifiable elements• There is known data that can be used to

distinguish and choose elements in the list

• Unidentifiable elements• No data in the elements in the list can be used

to distinguish or choose any of the elements


• Dropdown lists• Identifiable elements

• “If the list item has a bold letter preceding it, you can select the list item by pressing the letter.” (Manual # 100541)

Algorithm• Choose the list (usually an Alt-key combination)• Send( “c” ) or Send( “i” ) or Send( $choice )


No letters, yet works!

Note: first element is NOT default!


• Dropdown lists• Unidentifiable elements

Its ordinal position in the list is the only way to refer to an element

Algorithm• Choose the list (usually an Alt-key combination)• “Hard code” the list choices into the script• Based on the desired choice, use Home, End,

and the Up and Down arrows to move the highlight to the corresponding list element

• Send( “{ENTER}” ) to choose that element


END key moved to last element


• Fixed-field tables and variable fields (screen editor variations)

Algorithm• Click anywhere within the table or record• Select all of the data (Ctrl-a)• Copy the data into a string• Parse the string, looking for specific data

Getting item dataGetting item data


hex 20 = space


hex 09 = TAB (not spaces) hex 0d 0a = CRLF

(end of line)

Getting bib dataGetting bib data


hex 20 = space


hex 09 = TAB (not spaces)


hex 09 = TAB (not spaces)

hex 7c = subfield_delimiter

(pipe)

hex 0d 0a = CRLF (end of line)

Changing/deleting dataChanging/deleting data

• Fixed fields:• Click at end of fixed field table• Ctrl-Home to beginning of first fixed field• "tab to a field" approach


• Variable fields (top-down):• Click at end of fixed field table• Tab once to first variable field• "read a line at a time" approach (downward)

(using Home and Shift-End to get each line)


• Variable fields (bottom-up):• Ctrl-End to end of last variable field• Home to beginning of field• "read a line at a time" approach (upward)

(using Home and Shift-End to get each line)



To replace data in a variable field, rather than fiddle with insertions and such, it's sometimes easier to delete the field and then add the modified version as a new field

Next-to-last scripting “secret”Next-to-last scripting “secret”

• Using a script recorder is NOT cheating!

• Enables you to establish a starting point from which you can adapt as necessary

Final scripting “secret”Final scripting “secret”

• Reuse code written by yourself and others -- there’s no point in reinventing the wheel!

• 2009 Conference program web site:

FavFuncs.au3

REVIEW using Data Exchange

REVIEW using Data Exchange

REVIEW using Data ExchangeREVIEW using Data Exchange

OPTIONAL:

check title for correct window


OPTIONAL:

check pixels for correct screen


send keystroke


send keystroke (DOWN key)


send keystroke (END key)


send keystroke (ENTER key)


click mouse


send keystroke (TAB key)


send keystroke (SHIFT-HOME key)


send keystrokes (IP address)


send keystroke


send keystrokes:

Alt-u user_name Alt-p password


send keystroke


wait for window to appear

then wait for window to disappear


send keystroke


send keystroke (DOWN key)


send keystroke (identifier--“a” key)


click mouse


compare if this is desired filename

if not, send DOWN key, and compare next filename


send keystroke


wait for window to appear

then wait for window to disappear


send keystroke


send keystroke

alternative keystroke


OPTIONAL:

send keystroke

Why two parts in this talk?Why two parts in this talk?

• My vision:

To combine the Innovative system, AutoIt scripts, J scripts, and one’s local PC to create a powerful fusion of technologies for creating, managing, manipulating, and interpreting Innovative data

Some

Statistics

“Secrets”

PurposePurpose

• To introduce the J programming language as a powerful alternative and addition to Excel for handling exported Millennium data

What is “J”?What is “J”?

• An improved derivative of APL

• Invented in 1989 by Ken Iverson, who also invented APL in early 1960’s

• Availability (free): http://www.jsoftware.com


• Like APL, J operates on an entire array as a single entity rather than thinking of the array as a collection of separate elements

• J views nearly everything as an array


• Terms for arrays of increasing rank:• atom – a single element (rank 0)• list – one or more atoms (rank 1)• table – one or more lists (rank 2)• report – one or more tables (rank 3+)

Analogy of J to geometryAnalogy of J to geometry

• atom point (0 dimensions)

• list line (1 dimension)

• table plane (2 dimensions)

• report cube, hypercube, etc. (3+ dimensions)

Another J analogyAnother J analogy

• atom

• list

• table

• report


• Expanding upon the arithmetic symbols (+, -, etc.), J uses nearly all symbols (and some letters) on an ASCII keyboard to represent operations on arrays

• Each symbol also has 2 inflected forms, appending either a period or a colon


• Elements in a programming command can be any of these parts of language:• nouns• verbs• adverbs• conjunctions• gerunds


• There are only two basic command formats—everything else is a variation on these:

• verb noun (monadic - 1 arg)

• noun verb noun (dyadic - 2 args)

• Note: A noun argument can actually contain multiple arguments or parameters, if needed; a verb can be a J primitive or something you (or others) have written (because J is extensible)


• Both forms resolve to a noun, which can be substituted in a larger command:

( noun verb noun ) verb ( verb noun )

noun verb noun


• Often, but certainly not always, the righthand noun argument is data and the lefthand noun argument (if any) controls how the verb acts on the data


• The key difference in these noun arguments from other programming languages is that not only can they be single numeric or literal values but they can also be entire arrays!

• This is the main “secret” of J’s power


• A second “secret” of J’s power is that the explicit array looping needed in other languages is built into J’s commands

• This considerably simplifies the writing of scripts and helps mentally focus on the “big picture”


• Because J is so terse, often a whole program (or portions) can be written sideways rather than vertically!

• Here is an example of a whole program within a single line of code (frequencies of subject headings in a catalog):

sh7=.|:((|:~.(/:{)sh),(<&>#;.1~:(/:{)sh))


• Downsides:• Looks like “Martian”• Rather “geeky”• Huge amount of documentation, but very

difficult to find specific information• Steep learning curve may intimidate

beginners• Major paradigm shift in thinking


• Upsides:• Can do a lot with very little code• Availability of a very complete suite of

“assisting” applications, most written in J (plotting, forms and controls, spreadsheet, database, regular expressions, etc.)

• Size of arrays is limited only by amount of memory and disk space available

• So-called “boxed” arrays can contain mixed numeric and literal data

Examples of J primitivesExamples of J primitives

i. n create list of integers from 0 to n-1

|. reverse order of items in array

|: transpose (flip) rows/cols in table

/:~ sort items in list in ascending order

~. dedupe items in list

, append two sets of items

=: global assignment ( =. is local )

Simple J program/scriptSimple J program/script

br3=: ~. /:~ (br1,br2)

• br1 and br2 are two lists of bib rec numbers that have already been read from disk

• Meaning of script (executed right-to-left):• Append br2 to br1 to form a single large list• Sort this list• Dedupe the list• Assign (globally) the list to variable br3

More examples of JMore examples of J

• You can give your own names to verbs:

] revfile1=: 1 + i.10

1 2 3 4 5 6 7 8 9 10

] revfile2=: 2 3 5 8

2 3 5 8

remove=: -. NB. usually known as “not”

] result=: revfile1 remove revfile2 1 4 6 7 9 10


• You can create your own verbs: shfreq=: 3 : 0 NB. “x” is LH arg, “y” is RH arg of verb:

sh2=. (/:{) y NB. sorted list sh3=. # ;.1 ~: sh2 NB. freq’s sh4=. |: ~. sh2 NB. unique SH’s sh5=. < every sh3 NB. counts sh6=. |: (sh4,sh5) NB. combine SH’s & counts )

• Use new verb like any other J verb: freqlist=: shfreq sh NB. sh = file of SH’s


• You can use verbs that others have created:

updatewith is a verb written by someone proficient in J at my request

(see updatemasterfile.ijs on 2009 Conference program web site)

It appends a column of data to an existing (“master”) table, matching row data according to the keys in the first column of both files (this provides a workaround for III’s zero suppression in reports)


• Example of monthly statistics use:

load '~user\temp\updatemasterfile.ijs'

master=: jpath '~user\data\master.csv'

newdata=: jpath '~user\data\month01.csv'

master updatewith newdata

• Note: updatemasterfile.ijs script contains 2 verbs with 21 lines of code


• Example of use with patron checkouts:

load '~user\temp\updatemasterfile.ijs'

master=: jpath '~user\data\master.csv'

newdata=: jpath '~user\data\snapshot01.csv'

master updatewith newdata

• Note: this can be subsequently used to calculate circulation statistics between two or more “snapshot” dates


• It’s also possible to “GUI-fy” the previous simple scripts by having two dropcombo boxes of file directories that a staff member could choose from—without having to know or learn J themselves!

Appetite whetted for J?Appetite whetted for J?

• J has enough array and mathematical power to keep you learning for a lifetime

• J can be used independently of Excel

• J can be used in conjunction with Excel (there are also direct interfaces with Excel)

• I hope that I’ve interested you enough in J so that you can…

J J

Pick up the torch and go for it!

More informationMore information

• ---- AHML web site ---- (as long as it’s available)

http://www.ahml.info/oml

• Links: Scripts Using AutoIt

Scripts Using J (including my “Beginning J”)


• ---- WILIUG presentations ----

• More about AutoIt scripting (Jun 2006): http://www.wiliug.org/meetings/2006/6_06.html

• More about the J language (Jun 2009): http://www.wiliug.org/meet.html


• ------ Discussion groups ------

• Script language forums: http://www.autoitscript.com/forum/index.php? http://www.autohotkey.com/forum/ http://pgmacros.invisionzone.com/ (Macro Express)

• J language forums/groups: http://www.jsoftware.com/forums.htm http://groups.google.com/group/J-Programming

Email addressesEmail addresses

• After May 2009:

[email protected] or

[email protected]

Thanks for attending!Thanks for attending!

Q & AQ & A

© 2 0 0 9 b y H a r v e y E. H a h n S ome S ecrets of S uccessful S cripting and S tatistics h...

Documents

Transcript of © 2 0 0 9 b y H a r v e y E. H a h n S ome S ecrets of S uccessful S cripting and S tatistics h...