Indiana University QuakeSim Activities Marlon Pierce, Geoffrey Fox, Xiaoming Gao, Jun Ji, Chao Sun.
The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in...
-
Upload
diane-owens -
Category
Documents
-
view
223 -
download
0
Transcript of The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in...
![Page 1: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/1.jpg)
The Open Gateway Computing Environment: Experiences
Developing Tools for Scientific Communities in the Apache
Software Foundation Marlon Pierce
Indiana UniversityMarch 7, 2012
![Page 2: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/2.jpg)
March 7, 2012: The Competition for Your Attention
![Page 3: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/3.jpg)
IU Science Gateway Group Overview• IU Science Gateway Group members
– Marlon Pierce: Project Leader– Suresh Marru: Principal Software Architect– Raminder Singh, Chathura Herath, Yu Ma, Lahiru Gunathilake:
Senior team members– Research assistants and interns
• NSF SDCI funding of Open Gateway Computing Environments project – TACC (M. Dahan), SDSC (N. Wilkins-Diehr), SDSU (M. Thomas),
NCSA (S. Pamidighantam), UIUC (S. Wang), Purdue (C. Song), UTHSCSA (E. Brookes)
• We participate in two Apache incubators– Apache Rave: http://incubator.apache.org/rave/ – Apache Airavata: http://www.airavata.org
![Page 4: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/4.jpg)
Web user interfaces to Grids, Clouds, and other scientific resources.
Web user interfaces to Grids, Clouds, and other scientific resources.
Scientific workflow composition and execution.
Scientific workflow composition and execution.
![Page 5: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/5.jpg)
Compute ResourcesCompute Resources
Resource Middleware
Resource Middleware Cloud Interfaces Grid Middleware SSH & Resource
Managers
Computational Clouds Computational Grids
Gateway SoftwareGateway Software
User Interfaces
User Interfaces
Web/Gadget
Container
Web Enabled Desktop Applications
User Management
Auditing & Reporting
Fault Tolerance
Application Abstractions
Workflow System
Information ServicesMonitoring
Registry SecurityProvenance &
Metadata Management
Local Resources
Web/Gadget Interfaces
Gateway Abstraction Interfaces
Cyberinfrastructure Layers
Color Coding
Dependent resource provider components
Complimentary Gateway Components
OGCE Gateway Components
![Page 6: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/6.jpg)
1
CollaborationsCollaborating Team Scientific Field
GridChem (Sudhakar Pamidighantam, NCSA)
Computational Chemistry
ParamChem (Alex Mackerell, Sudhakar Pamidighantam, Micheal Sheetz et. al)
Molecular Sciences
WIYN Consortium One Degree Imager (Pat Knezek, NOAO)
Astronomy
OLAM (Craig Mattocks, University of Miami)
Atmospheric and Environmental Modeling
UltraScan (Borries Demeler, University of Texas Health Science Center)
Experimental Biophysics
LCCI (James Vary, Iowa State) Computational Nuclear Physics
Dark Energy Survey Simulation Working Group (August Evrard et. al)
Astrophysics, Astronomy
VLab (Renata Wentzcovitch, University of Minnesota)
Planetary Materials
![Page 7: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/7.jpg)
Key Problems for Science Gateway, Cyberinfrastructure Software
• Reusability– Reuse or write your own gateway software?
• Sustainability– The reason to reuse.– Cyberinfrastructure Software Sustainability and
Reusability: Report from an NSF-funded workshop• https://scholarworks.iu.edu/dspace/handle/2022/6701
• Governance– How are design decisions made?– Who decides if the software is suitable for release?– How do you handle contributions?– How do you add people to the development and project
management teams?
![Page 8: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/8.jpg)
OGCE Funds Software Lifecycle
![Page 9: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/9.jpg)
Governance: Open Community Software
• More than SourceForge, GitHub, Google Code, etc– Those provide excellent Web tools to help developers.
• Here we are concerned with community building.– Diverse community of developers increases probability
of reusability and sustainability– But diverse communities require governance
– Get governance right, and sustainability and reuse will follow.
![Page 10: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/10.jpg)
Some Open Model Examples in CI• NSF-funded CDIGS project
– http://confluence.globus.org/display/CDIGS/CDIGS+Home+Page• HUBzero Consortium, Sakai Foundation, Kuali Foundation
– Institutional level organizations • Eclipse Parallel Tools Platform
– Jay Alameda, NCSA: NSF SI2 funding to develop HPC tools workbench
– http://www.eclipse.org/ptp/• Enzo Project
– Excellent talk at TG11 by Prof. Brian O’Shea on their open community efforts
– http://enzo-project.org/• Apache Software Foundation
– OODT and TIKKA Data Management Projects at NASA JPL
![Page 11: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/11.jpg)
Two Apache Software Foundation Case Studies
Apache Rave and Airavata Incubators
![Page 12: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/12.jpg)
Apache Airavata
• Science Gateway software framework to– Compose, manage, execute, and monitor
computational workflows on Grids and Clouds– Web service abstractions to legacy command line-
driven scientific applications– Modular software framework to use as individual
components or as an integrated solution.• More Information
– Airavata Web Site: airavata.org– Developer Mailing Lists: airavata-
![Page 13: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/13.jpg)
Apache Airavata High Level Overview
![Page 14: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/14.jpg)
Example Workflow: Nuclear Physics
Courtesy of collaboration with Prof. James Vary and team, Iowa State
![Page 15: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/15.jpg)
1
Apache Rave
• Open Community Software for Enterprise Social Networking, Shareable Web Components, and Science Gateways
• Founding members:• Mitre Software• SURFnet• Hippo Software• Indiana University
• More information• Project Website: http://incubator.apache.org/rave/• Mailing List: [email protected]
![Page 16: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/16.jpg)
Gadget Dashboard View
Gadget Store View
![Page 17: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/17.jpg)
Rave Building Blocks
• Rave is implemented in JavaScript, Java with Spring MVC– Bean initialization specified in XML configuration files.– Inversion of Control makes it easy to swap out
implementations.– Disciplined MVC through Java annotations
• Builds on Apache Shindig and Wookie– Provide layout management, user management,
administration tools, production backend data systems, etc.
![Page 18: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/18.jpg)
Extending Rave for Science Gateways
• Two constraints– Must work out of the box– But must be flexible for developers to adapt it.
• Rave is designed to be extended.– Good design (interfaces, easily pluggable
implementations) and code organization are required.– It helps to have a diverse, distributed developer
community• How can you work on it if we can’t work on it?
• Rave is also packaged so that you can extend it without touching the source tree.
• GCE11 paper presented 3 case studies for Science Gateways
![Page 19: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/19.jpg)
Rave Extension General Steps
• Download and install Rave’s source– “mvn clean install” puts JARs, WARs, and POMs
into your local Apache Maven repository.– Only if building from a snapshot.
• Create a new Apache Maven project– You’ll need rave-portal-dependencies POM in your
<dependencies/>.– Include any configuration files that you would like
to modify.– Include the source code for your extensions.
![Page 20: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/20.jpg)
The Apache Way and Science Gateways
![Page 21: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/21.jpg)
Why Apache for Gateway Software?• Apache Software Foundation is a neutral playing field
– 501(c)(3) non-profit organization.– Designed to encourage competitors to collaborate on
foundational software.– Includes a legal cell for legal issues.
• Provides the social infrastructure for building communities.• Opportunities to collaborate with other Apache projects
outside the usual CI world.• Foundation itself is sustainable
– Incorporated in 1999– Multiple sponsors (Yahoo, Microsoft, Google, AMD, Facebook,
IBM, …)• Proven governance models
– Projects are run by Program Management Committees.– New projects must go through incubation.
![Page 22: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/22.jpg)
The Apache Way• Projects start as incubators with 1 champion and several mentors.
– Making good choices is very important• Graduation ultimately is judged by the Apache community.
– +1/-1 votes on the incubator list• Good, open engineering practices required
– DEV mailing list design discussions, issue tracking– Jira contributions– Important decisions are voted on
• Properly packaged code– Build out of the box– Releases are signed– Licenses, disclaimers, notices, change logs, etc.– Releases are voted
• Developer diversity– Three or more unconnected developers– Price is giving up sole ownership, replace with meritocracy
![Page 23: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/23.jpg)
Apache and Science Gateways• Apache rewards projects for cross-pollination.
– Connecting with complementary Apache projects strengthens both sides.
– New requirements, new development methods• Apache methods foster sustainability
– Building communities of developers, not just users– Key merit criterion
• Apache methods provide governance– Incubators learn best practices from mentors– Open, democratic procedures
– Processes for adding new committers and management– Ex: Releases are peer-reviewed and voted on.– All communications are archived.
![Page 24: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/24.jpg)
Apache Contributions Aren’t Just Software
• Apache committers and Project Management Committee members aren’t just code writers.
• Successful communities also include– Important users– Project evangelists – Content providers: documentation, tutorials– Testers, requirements providers, and constructive
complainers • Using Jira and mailing lists
– Anything else that needs doing.
![Page 25: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/25.jpg)
How To Get Involved
• Join the DEV mailing lists.• Grab the software and start complaining.• Post Jira tickets
– Add your patches to Jira if you want to solve a problem.
– Request a review– Frequent patch submission is the best way to get
voted in as a committer.
![Page 26: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/26.jpg)
![Page 27: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/27.jpg)
Case Study: GridShib and Community Credentials
• XSEDE Science Gateways use shared community credentials when accessing backend resources. – Many portal users map to one community
account.• GridShib adds attributes to grid credentials
– Gateway membership, originating IP address, user email, creation time, etc.
• For Rave, we’ll have to change the User service implementation to support this.
![Page 28: The Open Gateway Computing Environment: Experiences Developing Tools for Scientific Communities in the Apache Software Foundation Marlon Pierce Indiana.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649e035503460f94aef0fb/html5/thumbnails/28.jpg)
GridShib Step By Step• Install Rave in your Maven repo.• Create a Maven project with standard directory
layout for WAR packaging • Create a new user service (ComUserService) for
obtaining a community credential and adding GridShib attributes.
• Replace applicationContext-security.xml with your version
• In the XML, replace the default UserService with ComUserService.
• Place all GridShib resources in src/main/resources • Place web.xml in src/main/webapp/WEB-INF
– You’ll need an additional listener to get the IP address.