(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
-
Upload
accelrys -
Category
Technology
-
view
1.872 -
download
2
description
Transcript of (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
![Page 1: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/1.jpg)
(ATS3-PLAT07) Pipeline Pilot Tips, Tricks and Challenges
Ton van DaelenSr. Director, Platform Product Management
![Page 2: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/2.jpg)
The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.
![Page 3: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/3.jpg)
• Target audience– Protocol authors
• Personal productivity• Protocol deployment
• Many tips contributed by support• Mix of What’s New and Tips and Tricks
• Other relevant sessions– (ATS3-PLAT10) Win Matt's Money
Introduction
![Page 4: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/4.jpg)
1. Pro-client what’s new2. Protocol search3. Local short-cuts4. Recursion5. Keyboard shortcuts6. Design mode7. Component update8. Merge/join9. RTC subprotocols10. Debugging
10 Tips to Raise Your PP Game
![Page 5: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/5.jpg)
Protocol Search
• Tab– Move between tabs, retaining
search string
• Ctrl-Alt-F– Search utilities, prototypes,
internals– Be careful: it’s hidden for a reason!
• Search string– Tip: type first letters for keywords,
e.g.: “HTM Vie Mol”
Craig Shepherd - Support, San Diego
![Page 6: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/6.jpg)
Favorites Bar
• What– Folders– Most used
• Why– Find components quickly– Accelerate protocol
building
![Page 7: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/7.jpg)
Enhanced Tool Tips
• What– Tooltip explaining preventing a
protocol from running
• Why– Lower learning curve for novice users
![Page 8: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/8.jpg)
• What– Display processing time for each component (Ctrl-T)
• Absolute compute time (sec)• Compute time as percentage of total execution time
• Why– Identify protocol bottlenecks
Component Profiling
![Page 9: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/9.jpg)
PilotScript Context Sensitive Help
![Page 10: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/10.jpg)
• What– Multiple references to same
subprotocol within a protocol
• Why– Reduce protocol size– Re-use of subprotocol defintions
Local Shortcuts
![Page 11: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/11.jpg)
Design Mode
• What– Inspect data stream as
you build your protocol
• Why– Speed up protocol
development time >10x
![Page 12: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/12.jpg)
Design Mode
• Rapid protocol building and debugging– Monitor data in real-time– Immediately see affect of component and parameter changes
• Particularly powerful for Reporting and Imaging applications
![Page 13: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/13.jpg)
Using Design Mode
toggle design mode design on
Turn on design mode ...... puts data caches (“checkpoints”) at each (non-writer, non-viewer) component... after first run, data caches get filled , protocol may re-run from any of these... changes of components invalidate data caches... clicking on components shows data records in AutoWatch window
... From component context-menu: create individual windows; F8: on/off
Update data caches
![Page 14: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/14.jpg)
Design Mode: Statistics
View cache information
Throughput statistics
![Page 15: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/15.jpg)
Design Mode: Preferences
![Page 16: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/16.jpg)
Design Mode: Watch Window
Toggle Default View / Hierarchical View
Toggle Show / Hide all imagesFilter by data to view types
![Page 17: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/17.jpg)
Design Mode: Multiple Watch Windows
• Watch data at multiple stages in the pipeline• Add watch windows using F8 key
![Page 18: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/18.jpg)
![Page 19: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/19.jpg)
Protocol Recovery
• What– Keep copy of unsaved
protocol• Why
– Don’t lose work when client crashes
Saved in the Windows user’s temp directory, e.g.:C:\Users\tvandaelen\AppData\Local\Temp\scitegic\PipelinePilot\8.5.0.200 Default saving frequency is 5 minutes (value saved in registry): HKEY_CURRENT_USER\Software\SciTegic\Pipeline Pilot_8_9_1\Options\AutoSaveMinutes (If you don’t see it in your registry, open Options in Pro Client, click OK, then refresh reg key
![Page 20: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/20.jpg)
Pipeline Pilot Quick Reference Card
• Grab a copy• My favorites– CTRL-DblClick– CTRL-M– CTRL-SHIFT
I/P/F– F7– F10
Stephane Vellay - Support, Cambridge
![Page 21: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/21.jpg)
Looping in Pipelines
• DON’T DO IT!!!• But if you do, be very, very careful• It can work, but we don’t officially support it• Why? Datapipelining is about “data flow”.
Looping pipes is about “control flow”. Mixing these can be very problematic– e.g., a checkpoint in a loop will alter behavior– A generator inside a loop can be
“troublesome…”• Any looped protocol can be re-written
without loops• Control better exerted within components
(e.g., PilotScript), or across pipelines
Ian Kerman - Support, San Diego
![Page 22: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/22.jpg)
Looping
• Calculate factorials– Fac(3) = 3*2*1
• Make sure there is a exit point!
![Page 23: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/23.jpg)
Recursion
• Avoids looping• Again, make sure
your component exits
Factorial *= FacNumber; FacNumber--;
![Page 24: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/24.jpg)
Iteration
• Use Generate Empty Data to create data stream with fixed number of records
![Page 25: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/25.jpg)
Display Globals
![Page 26: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/26.jpg)
Component Update
• Useful for:– Component
upgrades– Regular protocol
building
Craig Shepherd - Support, Cambridge
![Page 27: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/27.jpg)
• Takes all streaming records with identical key values and collapses them into a single record
Merge Data
Key = ID
![Page 28: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/28.jpg)
Join Data From File
• For streaming records, finds related information in reference file (based on identical key values)
Key = IDID Exp1A 4B 4C 1D 3E 2F 1
ID ActivityA 4.1B 2.1C -0.5A 4.3Q 3.0
ID Activity Exp1A 4.1 4B 2.1 4C -0.5 1A 4.3 4
ID ActivityQ 3.0
These records not seen in the output
Reference File
![Page 29: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/29.jpg)
Merge All Data
• Merges properties of all input data records into a single output data record– Performs faster than Merge Data for the special case where all
input records are to be merged into one
![Page 30: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/30.jpg)
Merge/Group/Pivot – Data Are Sorted
• New parameter Data Are Sorted
• Specifies whether the input data are already sorted by the Merge property. If so, then the merge can be done much faster with less memory impact
![Page 31: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/31.jpg)
Quick Reference Guides
![Page 32: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/32.jpg)
RTC Subprotocols
![Page 33: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/33.jpg)
• In a RTC subprotocol, all internal components are executed for all incoming data records
• Use sparingly…• Relevant when there are initialization and/or finalization
components in subprotocol• Often used on “grouped” data
– Group prior to RTC SP > Ungroup in RTC SP > process
• In 8.0: Use a category/tag to auto-group data records and apply the RTC subprotocol
Run to Completion (RTC) Subprotocols
![Page 34: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/34.jpg)
Ordinary data flow
Reader
Reader
Group Ungroup MCSS
Group Data By Tag aggregates the dataand “boxes” it
When all of the data arrives, the MCSScomponent acts on the data
Data moves as far to the right as possible
![Page 35: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/35.jpg)
Subprotocol
Data flow through a subprotocol
Reader
Reader
Group Ungroup MCSS
Subprotocol is a way to visually hide work – No change in behavior
![Page 36: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/36.jpg)
Process is repeated independently for each group
RunToCompletion Subprotocol
Data flow with RunToCompletion
Reader
Reader
Group Ungroup MCSS
Data is aggregated and partitioned by the“Group data by tag” component
MCSS component “sees” and actson only one box of data
Inside a RunToCompletion subprotocol, asingle data record (a “box”) is unpacked
![Page 37: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/37.jpg)
Run to Completion – Simplified Interface
• Grouping / ungrouping obsolete in most cases
• Grouping behavior set as subprotocol parameter
![Page 38: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/38.jpg)
Debugging Tips
![Page 39: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/39.jpg)
Displaying Globals and Data Records
What’s my Data?
Don’t know what the syntax is for a property name?
Find out: Put in a viewer!
![Page 40: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/40.jpg)
Why did I close that output Window so quickly?
• Never fear! Recover results from the Jobs window.
![Page 41: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/41.jpg)
Checkpoints and Watch Windows
• Method to cache the data flowing through a pipeline
• Re-run a job from a specific location in the pipeline
• Skip time consuming steps when debugging
• View the contents of a checkpoint cache using watch windows
![Page 42: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/42.jpg)
Checkpoints and Watch Windows (cont.)
• Watch can be added on to any component containing a cache
• Max of 3 Watch windows• Auto-watch displays currently selected
component cache• Multiple views
– Default View– Hierarchical View– Molecular Images View
• Data Type Filtering– All Data– Pass Data – Fail Data– Global Variables
![Page 43: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/43.jpg)
• Provides a method to debug protocols at run time
• Displays a Debug Messages window that shows the results from debug message commands– DebugMessage: Displays a debug text message in the Debug Messages
window– DebugMessageError: Displays a debug text message (in red) in the Debug
Messages window– DebugBreak: Pauses protocol execution and displays a message in a dialog
box. Processing continues when you click the Resume button
• Commands available in Pilotscript, Java and Perl
• Debug Parameters Window– Not limited to components that use PilotScript, Java or Perl expressions.– Allows the addition of debug statements that test the outputs for any
component type
Running with Debugging
![Page 44: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/44.jpg)
• Want to read in a data file quickly?– Just drag any file into an open protocol window to
automatically invoke its reader type (SD, Mol2, PDB, etc)– Drag any file onto an existing Reader to append that file to
the data stream• Want to use a protocol quickly that you get in an email?– Drag any protocol (e.g. email attachments, protocols in other
directories, etc) onto the protocol window to open them
Rapidly accessing files and protocols
![Page 45: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/45.jpg)
• Lots of new productivity features in 8.0 and 8.5• Resources– Community forum
• Share your insights!
– Support hotline– Training
Summary
![Page 46: (ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges](https://reader033.fdocuments.net/reader033/viewer/2022061210/549079a8b47959b9458b4605/html5/thumbnails/46.jpg)
The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.
For more information on the Accelrys Tech Summits and other IT & Developer information, please visit:https://community.accelrys.com/groups/it-dev