Winners and Losers: Ranking Crystals from Diffraction Images Angela R. Criswell Automation...

$: Winners and Losers: Ranking Crystals from Diffraction Images Angela R. Criswell Automation Scientist.$
Winners and Losers:Ranking Crystals from

Diffraction Images

Angela R. CriswellAngela R. CriswellAutomation ScientistAutomation Scientist

ACTOR InstallationsACTOR Installations• Pharmaceutical Companies (11)Pharmaceutical Companies (11)

Abbott Laboratories (Chicago, IL) Astex Technology (UK) AstraZeneca (UK) Aventis (Frankfurt) BMS (Princeton, NJ) Exelixis (San Francisco, CA) Merck (West Point, PA) Novartis (Basel, Switzerland) Novartis (Cambridge, MA) Pfizer (St. Louis, MO) Schering-Plough Research Inst. (NJ)

• Structural Genomics Groups (3)Structural Genomics Groups (3) SGC – Oxford (UK) University of Georgia University of Toronto

• Beamlines (2)Beamlines (2) Daresbury Laboratory (UK) IMCA-CAT (APS)

• Future Installations (4)Future Installations (4) 2 additional beamlines (SLS, Diamond) 1 pharmaceutical company

• AGENT Installations (3)AGENT Installations (3) ActiveSight (San Diego, CA) 2 future pharmaceutical sites

High Throughput OptimizationHigh Throughput Optimization

• Automate the processesAutomate the processes Crystallization robots Sample mounting robots Automated structure solution

• Increase robustness for automated processesIncrease robustness for automated processes Hardware and software improvements Sample tracking methods and database management

• Ever increasing complexityEver increasing complexity Incorporate intelligence and examine success/failure.

Heuristic and learning methods

Remote access and control of automated processes VNC and mail-in crystallography

Diffraction improvement by controlled hydration Free-mounting system (Proteros)

Crystal Ranking: An EvolutionCrystal Ranking: An Evolution

• Do I have another crystal??Do I have another crystal??

• Is the crystal twinned?Is the crystal twinned?• How far does the crystal diffract? How far does the crystal diffract? • Are there ice rings?Are there ice rings?• Do peaks have a decent spot shapes? Do peaks have a decent spot shapes? • Can I assign a unit cell for the sample?Can I assign a unit cell for the sample?• What are the unit cell dimensions and space group?What are the unit cell dimensions and space group?

How do CrystallographersHow do CrystallographersRank Crystals??Rank Crystals??

• I/sig(I) analysis is not sufficientI/sig(I) analysis is not sufficient• Single image is probably not sufficientSingle image is probably not sufficient

Crystal Ranking EffortsCrystal Ranking Efforts

• d*TREK d*TREK (Rigaku/MSC - Pflugrath)(Rigaku/MSC - Pflugrath) automatic indexing, ranking, strategy, integration, scaling

• DISTL and LABELIT DISTL and LABELIT (SSRL & LBNL)(SSRL & LBNL) Automatic ranking and indexing, data processing

• DNA DNA (SPINE)(SPINE) Automatic ranking and indexing

• CrySis CrySis (Brookhaven – Bernston, Stojanoff, and Takai)(Brookhaven – Bernston, Stojanoff, and Takai) ranking with neural network trained with 500 diff images

• BEST BEST (EMBL – Popov)(EMBL – Popov) Data collection strategy based upon statistic modeling

SpamAssassinSpamAssassin

Email SCORE: Advertisement for SuperBowl Celebration EventEmail SCORE: Advertisement for SuperBowl Celebration Event• No. hits=3.9 Required=4.0 No. hits=3.9 Required=4.0

tests=HTML_60_70HTML_FONTCOLOR_REDHTML_FONTCOLOR_UNSAFEHTML_FONT_INVISIBLEHTML_MESSAGEHTTP_ESCAPED_HOSTHTTP_EXCESSIVE_ESCAPESLINES_OF_YELLING

• Performs cursory header analysisPerforms cursory header analysis: : spots emails that try to mask their identitiesspots emails that try to mask their identities

• Performs in-depth text analysisPerforms in-depth text analysis: : spam mails often have a characteristic style (to put it politely)spam mails often have a characteristic style (to put it politely) characteristic disclaimers and lots of !!!!! webpage links

• Enables blacklistingEnables blacklisting: : block email from existing blacklist sitesblock email from existing blacklist sites

• AdaptiveAdaptivelearns to recognize spam based upon user scores and amend blacklistslearns to recognize spam based upon user scores and amend blacklists

Strategic Ranking GoalsStrategic Ranking Goals

• Incorporate image analysis tools aloneIncorporate image analysis tools alone Diffraction limits Bragg peak intensities

Background radiation Ice ring identification – strong and diffuse

• Incorporate indexing and refinement resultsIncorporate indexing and refinement results Spot shape Lattice quality Spot prediction analysis (discriminates twinned from non-twinned

crystals)• Incorporate Comparative analysisIncorporate Comparative analysis

Between samples (rank comparisons) Images collected for same sample (different crystal orientations) Automatic exposure time determination

Rules 1 and 2Rules 1 and 2

• Divide image into 10 resolution bins.

• Ignore lowest 3 bins.

• Analyze 7 highest resolution shells• # reflns / shell• S:N of reflns / shell

Rule 3: Spot SharpnessRule 3: Spot Sharpness

• calculated for every peakcalculated for every peakoutput = avg 2(A/B)

A = peak max position – peak center position A = peak max position – peak center position

xx11 x x22

B = ( B = ( ΔΔx 2 + x 2 + ΔΔy 2 )1/2y 2 )1/2B is the effective diameter of the peak.

Rules 4 – 5: Ice Ring DetectionRules 4 – 5: Ice Ring Detection

• Step 1: filter out peaks from imagesStep 1: filter out peaks from images• Step 2: bin pixels by 2Step 2: bin pixels by 2θθ• Step 3: for each bin, sum pixel intensitiesStep 3: for each bin, sum pixel intensities

Example plot:

0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05

2Theta

Pix

el I

nte

nsi

ty

Lysozyme 2_05Lysozyme 2_05rank = 202rank = 202

0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05 45.05 50.05

Resolution (2theta)

Pix

el In

tens

ity


0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05 45.05 50.05

Resolution (2theta)

Pix

el In

tens

ity

Rules 6 - 11Rules 6 - 11

6. IndexingAward for percentage of indexed spots

7. RefinementPenalty based upon RMSMM residual

8. MosaicityPenalty based upon refined mosaicity

9. Refinement CoverageAward for percentage of accepted reflections in prediction list

10. PredictionRe-evaluate highest 7 resolution shells based upon number of found

spots that match predicted reflection list

11. Refined Reflection ResolutionRe-evaluate highest 7 resolution shells based upon the signal-to-noise

ratio of predicted reflections

Rule 1: Spot count in resolution shells (found spots)Rule 1: Spot count in resolution shells (found spots) Rule 2: I/Sigma in resolution shells (found spots)Rule 2: I/Sigma in resolution shells (found spots) Rule 3: Spot sharpnessRule 3: Spot sharpness Rule 4: Strong ice ringsRule 4: Strong ice rings Rule 5: Diffuse ice ringsRule 5: Diffuse ice rings Rule 6: Percentage of spots indexedRule 6: Percentage of spots indexed Rule 7: RMS residual after refinementRule 7: RMS residual after refinement Rule 8: MosaicityRule 8: Mosaicity Rule 9: Percentage of spots refinedRule 9: Percentage of spots refined Rule 10: Spot count in resolution shells (predicted and found spots)Rule 10: Spot count in resolution shells (predicted and found spots) Rule 11: I/Sigma in resolution shells (predicted and found spots)Rule 11: I/Sigma in resolution shells (predicted and found spots)

Sample / Rules 1 2 3 4 5 6 7 8 9 10 11 TotalSample / Rules 1 2 3 4 5 6 7 8 9 10 11 Total

L:\Images\lyso101_????.osc 1 70 60 -1 -10 0 50 -17 -20 28 70 62 292L:\Images\lyso101_????.osc 1 70 60 -1 -10 0 50 -17 -20 28 70 62 292

Ranking ResultsRanking Results

Sample Group #1Sample Group #1Tests with Lysozyme crystalsTests with Lysozyme crystals


0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05 45.05 50.05

Resolution (2theta)

Pix

el In

tens

ity

------------------------------------------------------------------------------- Category Points Cumul ------------------------------------------------------------------------------- >=5 reflns found in 2nd shell (1.79-1.86)Å 10 10 >=5 reflns found in 3rd shell (1.86-1.94)Å 10 20 >=5 reflns found in 4th shell (1.94-2.04)Å 10 30 >=5 reflns found in 5th shell (2.04-2.17)Å 10 40 >=5 reflns found in 6th shell (2.17-2.34)Å 10 50 >=5 reflns found in 7th shell (2.34-2.58)Å 10 60 I/sig == 44.8 in 2nd found shell (1.79-1.86)Å 7 67 I/sig == 56.8 in 3rd found shell (1.86-1.94)Å 9 76 I/sig == 60.1 in 4th found shell (1.94-2.04)Å 10 86 I/sig == 67.7 in 5th found shell (2.04-2.17)Å 10 96 I/sig == 74.2 in 6th found shell (2.17-2.34)Å 10 106 I/sig == 89.7 in 7th found shell (2.34-2.58)Å 10 116 Penalty for spot sharpness of 0.06 -1 115 Penalty for strong ring (2.82%) near resln. 3.513 -10 105 Penalty for diffuse ring (0.70%) near resln. 3.943 -5 100 Indexed 404 spots, or 75% of all spots used in indexing 74 174 Penalty for RMS residual value of 0.164 -16 158 Penalty for Mosaicity value of 0.4 -19 139 Refined 44 spots, or 4% of all predictions 3 142 >=5 reflns predicted and found in 5th shell (2.04-2.17)Å 10 152 >=5 reflns predicted and found in 6th shell (2.17-2.34)Å 10 162 >=5 reflns predicted and found in 7th shell (2.34-2.58)Å 10 172 I/sig == 77.7 in 5th predicted and found shell (2.04-2.17)Å 10 182 I/sig == 80.8 in 6th predicted and found shell (2.17-2.34)Å 10 192 I/sig == 94.5 in 7th predicted and found shell (2.34-2.58)Å 10 202 ------------------------------------------------------------------------------- Cumulative 202


0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05 45.05 50.05

Resolution (2theta)

Pix

el In

tens

ity

------------------------------------------------------------------------------- Category Points Cumul ------------------------------------------------------------------------------- >=5 reflns found in 2nd shell (1.79-1.86)Å 10 10 >=5 reflns found in 3rd shell (1.86-1.94)Å 10 20 >=5 reflns found in 4th shell (1.94-2.04)Å 10 30 >=5 reflns found in 5th shell (2.04-2.17)Å 10 40 >=5 reflns found in 6th shell (2.17-2.34)Å 10 50 >=5 reflns found in 7th shell (2.34-2.58)Å 10 60 I/sig == 49.8 in 2nd found shell (1.79-1.86)Å 8 68 I/sig == 47.0 in 3rd found shell (1.86-1.94)Å 7 75 I/sig == 52.8 in 4th found shell (1.94-2.04)Å 8 83 I/sig == 65.7 in 5th found shell (2.04-2.17)Å 10 93 I/sig == 69.9 in 6th found shell (2.17-2.34)Å 10 103 I/sig == 86.8 in 7th found shell (2.34-2.58)Å 10 113 Penalty for spot sharpness of 0.10 -1 112 Penalty for strong ring (2.78%) near resln. 3.555 -10 102 Penalty for diffuse ring (0.55%) near resln. 3.943 -5 97 Indexed 342 spots, or 56% of all spots used in indexing 56 153 Penalty for RMS residual value of 0.182 -18 135 Penalty for Mosaicity value of 0.3 -15 120 Refined 24 spots, or 2% of all predictions 2 122 >=5 reflns predicted and found in 4th shell (1.94-2.04)Å 10 132 >=5 reflns predicted and found in 5th shell (2.04-2.17)Å 10 142 >=5 reflns predicted and found in 6th shell (2.17-2.34)Å 10 152 I/sig == 44.4 in 4th predicted and found shell (1.94-2.04)Å 7 159 I/sig == 87.2 in 5th predicted and found shell (2.04-2.17)Å 10 169 I/sig == 67.0 in 6th predicted and found shell (2.17-2.34)Å 10 179 ------------------------------------------------------------------------------ Cumulative 179


0.05 5.05 10.05 15.05 20.05 25.05 30.05 35.05 40.05 45.05 50.05

Resolution (2theta)

Pix

el In

tens

ity

------------------------------------------------------------------------------- Category Points Cumul ------------------------------------------------------------------------------- >=5 reflns found in 3rd shell (1.86-1.94)Å 10 10 >=5 reflns found in 4th shell (1.94-2.04)Å 10 20 >=5 reflns found in 5th shell (2.04-2.17)Å 10 30 >=5 reflns found in 6th shell (2.17-2.34)Å 10 40 >=5 reflns found in 7th shell (2.34-2.58)Å 10 50 I/sig == 54.8 in 3rd found shell (1.86-1.94)Å 9 59 I/sig == 55.3 in 4th found shell (1.94-2.04)Å 9 68 I/sig == 64.3 in 5th found shell (2.04-2.17)Å 10 78 I/sig == 72.1 in 6th found shell (2.17-2.34)Å 10 88 I/sig == 86.1 in 7th found shell (2.34-2.58)Å 10 98 Penalty for spot sharpness of 0.07 -1 97 Penalty for strong ring (2.64%) near resln. 4.162 -10 87 Penalty for strong ring (2.05%) near resln. 3.875 -10 77 Penalty for strong ring (1.84%) near resln. 3.434 -10 67 Penalty for strong ring (6.76%) near resln. 2.139 -10 57 Penalty for strong ring (7.87%) near resln. 1.975 -10 47 Penalty for strong ring (4.78%) near resln. 1.875 -10 37 Indexed 305 spots, or 58% of all spots used in indexing 58 95 Penalty for RMS residual value of 0.121 -12 83 Penalty for Mosaicity value of 0.4 -18 65 >=5 reflns predicted and found in 5th shell (2.04-2.17)Å 10 75 >=5 reflns predicted and found in 6th shell (2.17-2.34)Å 10 85 >=5 reflns predicted and found in 7th shell (2.34-2.58)Å 10 95 I/sig == 57.8 in 5th predicted and found shell (2.04-2.17)Å 9 104 I/sig == 61.2 in 6th predicted and found shell (2.17-2.34)Å 10 114 I/sig == 103.3 in 7th predicted and found shell (2.34-2.58)Å 10 124 ------------------------------------------------------------------------------- Cumulative 124


------------------------------------------------------------------------------- Category Points Cumul ------------------------------------------------------------------------------- >=5 reflns found in 5th shell (2.25-2.39)Å 10 10 >=5 reflns found in 6th shell (2.39-2.57)Å 10 20 >=5 reflns found in 7th shell (2.57-2.83)Å 10 30 I/sig == 15.7 in 5th found shell (2.25-2.39)Å 2 32 I/sig == 19.5 in 6th found shell (2.39-2.57)Å 3 35 I/sig == 22.9 in 7th found shell (2.57-2.83)Å 3 38 Penalty for spot sharpness of 0.10 -1 37 Penalty for strong ring (1.09%) near resln. 4.031 -10 27 Indexed 242 spots, or 57% of all spots used in indexing 57 84 Penalty for RMS residual value of 0.086 -8 76 Penalty for Mosaicity value of 0.5 -20 56 Refined 186 spots, or 19% of all predictions 18 74 >=5 reflns predicted and found in 5th shell (2.25-2.39)Å 10 84 >=5 reflns predicted and found in 6th shell (2.39-2.57)Å 10 94 >=5 reflns predicted and found in 7th shell (2.57-2.83)Å 10 104 I/sig == 17.6 in 5th predicted and found shell (2.25-2.39)Å 2 106 I/sig == 19.7 in 6th predicted and found shell (2.39-2.57)Å 3 109 I/sig == 22.4 in 7th predicted and found shell (2.57-2.83)Å 3 112 ------------------------------------------------------------------------------- Cumulative 112

Effect of Indexing on Rank Values Effect of Indexing on Rank Values

-150

-100

-50

0

50

100

150

200

250

300

350 Rank - All RulesRank - Rules 1,2 only

Score VariabilityScore VariabilityRank Values vs. Exposure TimeRank Values vs. Exposure Time

Images / Rules 1 2 3 4 5 6 7 8 9 10 11 TotalImages / Rules 1 2 3 4 5 6 7 8 9 10 11 Total

Thaumatin – 5 sec/0.5º: RThaumatin – 5 sec/0.5º: Rmerge merge = 12.9 % (32.5 %)= 12.9 % (32.5 %)thau3 501,561 60 22 -2 -20 0 56 -5 -9 19 70 23 214thau3 501,561 60 22 -2 -20 0 56 -5 -9 19 70 23 214thau3 501 60 22 -2 -20 0 58 -5 -12 14 60 21 196thau3 501 60 22 -2 -20 0 58 -5 -12 14 60 21 196thau3 545 50 18 -3 -20 0 55 -5 -6 24 50 16 179thau3 545 50 18 -3 -20 0 55 -5 -6 24 50 16 179thau3 590 60 28 -3 -20 0 55 -6 -10 18 60 22 204thau3 590 60 28 -3 -20 0 55 -6 -10 18 60 22 204thau3 626 50 22 -3 -20 0 59 -6 -7 20 70 26 211thau3 626 50 22 -3 -20 0 59 -6 -7 20 70 26 211



197.5

221

260

Images / Rules 1 2 3 4 5 6 7 8 9 10 11 TotalImages / Rules 1 2 3 4 5 6 7 8 9 10 11 Total

VariMax-HR : RVariMax-HR : Rmerge merge = 2.9 % (22.3 %) = 2.9 % (22.3 %) LYS0503_screen 1-2 70 46 -1 -10 -5 51 -18 -13 46 70 42 278LYS0503_screen 1-2 70 46 -1 -10 -5 51 -18 -13 46 70 42 278LYS0503_screen 1 70 46 -1 -10 -5 54 -15 -11 50 70 41 289LYS0503_screen 1 70 46 -1 -10 -5 54 -15 -11 50 70 41 289LYS0503_screen 2 70 44 -1 -10 0 56 -18 -14 44 70 41 282LYS0503_screen 2 70 44 -1 -10 0 56 -18 -14 44 70 41 282LYS0503_ 1 70 46 -1 0 -5 56 -17 -12 48 70 42 297LYS0503_ 1 70 46 -1 0 -5 56 -17 -12 48 70 42 297LYS0503_ 45 70 46 -1 -10 0 57 -15 -13 45 70 42 291LYS0503_ 45 70 46 -1 -10 0 57 -15 -13 45 70 42 291LYS0503_ 90 70 46 -1 -10 -5 57 -16 -12 47 70 42 288LYS0503_ 90 70 46 -1 -10 -5 57 -16 -12 47 70 42 288LYS0503_ 116 70 46 -1 -10 -5 57 -16 -12 49 70 40 288LYS0503_ 116 70 46 -1 -10 -5 57 -16 -12 49 70 40 288

VariMax-HR : RVariMax-HR : Rmerge merge = 2.8 % (15.0 %)= 2.8 % (15.0 %)LYS0503_screen 1-2 70 57 -1 -10 0 56 -23 -18 39 70 57 297LYS0503_screen 1-2 70 57 -1 -10 0 56 -23 -18 39 70 57 297LYS0503_screen 1 70 58 -1 0 -15 57 -23 -17 42 70 57 298LYS0503_screen 1 70 58 -1 0 -15 57 -23 -17 42 70 57 298LYS0503_screen 2 70 57 -1 -10 0 59 -23 -17 42 70 57 304LYS0503_screen 2 70 57 -1 -10 0 59 -23 -17 42 70 57 304LYS0503_ 1 70 57 -1 -10 0 58 -21 -17 43 70 56 305LYS0503_ 1 70 57 -1 -10 0 58 -21 -17 43 70 56 305LYS0503_ 45 70 58 -1 -10 0 57 -21 -17 46 70 56 308LYS0503_ 45 70 58 -1 -10 0 57 -21 -17 46 70 56 308LYS0503_ 90 70 58 -1 -10 -5 55 -22 -18 39 70 57 293LYS0503_ 90 70 58 -1 -10 -5 55 -22 -18 39 70 57 293LYS0503_ 116 70 57 -1 0 -5 57 -22 -14 47 70 55 314LYS0503_ 116 70 57 -1 0 -5 57 -22 -14 47 70 55 314

Score VariabilityScore VariabilityData sets collected with VariMax opticsData sets collected with VariMax optics

291

305

What Have We Learned?What Have We Learned?

• Signal-to-noise is predominant factor in current d*TREK releaseSignal-to-noise is predominant factor in current d*TREK release This is intentional! Should it be? Each of the 11 rules have independent parameters that can be adjusted to

optimize for your case• Image processing adds domino effect to rankingImage processing adds domino effect to ranking

Better refinement, higher rank Lower mosaicity, higher rank Fewer twin spots, higher rank

• Spot sharpness analysis is not robustSpot sharpness analysis is not robust Incorporate graph theory

• Potential PitfallsPotential Pitfalls Weak diffractors

lowest 3 resolution bins should not excluded from spot analysis Image Header Accuracies Anisotropy

Need images at multiple angles These effects become effectively ‘averaged’ across images

Merohedral twinning

Recent d*TREK ImprovementsRecent d*TREK Improvements

• Don’t ignore lowest resolution bins Don’t ignore lowest resolution bins • Image Header AccuraciesImage Header Accuracies

Command line override

• AnisotropyAnisotropy Incorporated anisotropy check and another rule

Rank each image, calculate average and ESD Apply penalty as multiple of ESD

• Data Collection Strategy improvementsData Collection Strategy improvements Automatic exposure time calculation (using ‘intelligent’ algorithm) Optimize detector space for diffraction resolution Multiple scan strategy, if possible

AcknowledgementsAcknowledgements

Russ AthayRuss AthayRobert BolotovskyRobert BolotovskyJoseph D. FerraraJoseph D. FerraraThad NiemeyerThad NiemeyerKaren OperstenyKaren OperstenyJ.W. PflugrathJ.W. Pflugrath

Winners and Losers: Ranking Crystals from Diffraction Images Angela R. Criswell Automation...

Documents

Transcript of Winners and Losers: Ranking Crystals from Diffraction Images Angela R. Criswell Automation...