XNAT Tuning & Monitoring
-
Upload
john-paulett -
Category
Technology
-
view
2.036 -
download
1
description
Transcript of XNAT Tuning & Monitoring
![Page 2: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/2.jpg)
Overview
Share NRG’s experiences running a large XNAT
installation, including methods for tuning,
testing, and monitoring the application.
![Page 3: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/3.jpg)
Plan
1. Sample XNAT Architecture
2. Hardware “Recommendations”
3. Monitoring XNAT
4. Performance Testing Tools
5. Tuning XNAT
![Page 4: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/4.jpg)
XNAT SCALES!
![Page 5: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/5.jpg)
XNAT Scales
From a single study with dozens of scans
To hundreds of studies, including large, multi-
site studies
![Page 6: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/6.jpg)
Central Neuroimaging Data Archive
(CNDA)
Flagship XNAT installation at Washington
University (WUSTL)
As of June 2010:
– 500 studies
– 8000 subjects
– 11k imaging sessions
– 9 TB of data
![Page 7: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/7.jpg)
CNDA Architecture
![Page 8: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/8.jpg)
CNDA Architecture
• 1x Kemp load balancer
– SSL acceleration
• 1x Quad-core Xeon, 16GB RAM: PostgreSQL
8.3
• 2x Dual-core Xeon, 4GB RAM (one in
standby): Tomcat 5.5 & DicomServer
• BlueArc NAS
• Multiple Sun Grid Engine nodes
![Page 9: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/9.jpg)
FUTURE ARCHITECTURE
![Page 10: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/10.jpg)
Future Architecture
Reduce single points of failure
– Standby Kemp load balancer
– PostgreSQL Warm Standby
– Actively load balance Tomcat
– Archival storage SAN
Use new super computer at WUSTL
Improve ability to upgrade without downtime
![Page 11: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/11.jpg)
Future Architecture
![Page 12: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/12.jpg)
“RECOMMENDED” HARDWARE
![Page 13: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/13.jpg)
Grow into your Architecture
Get single good server
– Multicore with 4-16GB RAM (better than
central.xnat.org)
– Consider your archive’s future size & location
When you outgrow:
– Buy a more powerful machine for PostgreSQL
– Leave Tomcat on first server
![Page 14: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/14.jpg)
MONITORING XNAT
![Page 15: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/15.jpg)
![Page 16: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/16.jpg)
Pingdom
• World-wide tests for site availability &
response time
• SMS & Email alerts when sites are
unavailable
![Page 17: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/17.jpg)
Munin
PostgreSQL, Tomcat, & Linux metrics over time
– Memory, CPU, queries, requests, etc.
![Page 18: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/18.jpg)
![Page 19: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/19.jpg)
![Page 20: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/20.jpg)
Monit
Active process monitoring & management
Define criteria for emailing alerts & restarting
processes
– CPU, memory thresholds
– Connection failures (check web services)
![Page 21: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/21.jpg)
PERFORMANCE TESTING TOOLS
![Page 22: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/22.jpg)
JMeter
Generate load & analyze throughput
Complex HTTP transactions
![Page 23: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/23.jpg)
JMeter
https://svn.apache.org/repos/asf/jakarta/jmeter/
![Page 24: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/24.jpg)
YourKit Profiling
Lower level debugging tool
Memory & CPU profiling
Hunt down memory leaks & code hot spots
Can instrument in production server
![Page 25: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/25.jpg)
http://jira.atlassian.com/browse/JRA-12524
![Page 26: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/26.jpg)
TUNING XNAT
![Page 27: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/27.jpg)
On Tuning
Tuning results dependent on many variables, what worked in one case may not work universally
XNAT is a complex system! Some parts are CPU bound, others are memory bound, and others are bandwidth bound
General rule: faster CPUs + more RAM + bigger network pipe = faster XNAT
![Page 28: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/28.jpg)
On Tuning
1. Find something that is “slow”
2. Quantify slowness
3. Tune
4. Quantify improvement
5. Go to #1
![Page 29: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/29.jpg)
PostgreSQL Tuning
PostgreSQL 8.3 has serious performance
improvements
Put PostgreSQL and Tomcat on separate
machines
– Get powerful database machine
– PostgreSQL can take advantage of multiple
cores & lots of RAM
![Page 30: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/30.jpg)
postgresql.conf
Default settings designed for legacy machines
Increase available memory. Allows query planner to do more work in RAM and less on disk
Increase max connections
Tweak kernel settings to allow access to more memory
![Page 31: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/31.jpg)
Tomcat Tuning
Increase available memory & use “server VM”
catalina.sh:
JAVA_OPTS="$JAVA_OPTS "-XX:MaxPermSize=256m" "-XX:PermSize=256m" "-mx1512m" "-server"
![Page 32: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/32.jpg)
Tomcat Tuning
Increase connections & threads in server.xml
– At this point, consider load balancing between
multiple Tomcat servers
![Page 33: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/33.jpg)
XNAT Tuning
Upgrade to XNAT 1.4!
Increase MaxConnections to database in WEB-INF/conf/InstanceSettings.xml
– Set in line with PostgreSQL’s max_connections
![Page 34: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/34.jpg)
Tools Mentioned
Google Analytics: Free
Pingdom: Monthly subscription. One site free
Munin: Open Source
Monit: Open Source
JMeter: Open Source
YourKit: Commercial. Time-limited trial & free
for open source
![Page 35: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/35.jpg)
Questions & Your Experiences?
http://www.xnat.org/XNAT+2010+Workshop+-
+Tuning,+Optimization,+Monitoring
![Page 36: XNAT Tuning & Monitoring](https://reader034.fdocuments.net/reader034/viewer/2022050919/54582538b1af9fc0638b5485/html5/thumbnails/36.jpg)