Creating a developer pipeline by teaching gateway technologies
-
Upload
marlon-pierce -
Category
Technology
-
view
60 -
download
0
Transcript of Creating a developer pipeline by teaching gateway technologies
![Page 1: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/1.jpg)
Welcome to the SGCI Webinar!• We will be starting shortly.
• Your audio has been muted, and you are encouraged to turn off your video during the presentation.• Controls for these are near the bottom of
the right-side control panel for BlueJeans.
• You may submit questions at any time using Chat, and the moderator will share them with the presenter when appropriate.
• This presentation will be recorded and slides will be posted.
1
![Page 2: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/2.jpg)
Creating a developer pipeline by teaching
gateway technologiesMarlon Pierce, Suresh Marru
Science Gateways Research Center, Indiana [email protected], [email protected]
![Page 3: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/3.jpg)
http://sgrc.iu.edu
![Page 4: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/4.jpg)
The Pipeline Problem
Finding and retaining people to work on science gateways is tough.
![Page 5: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/5.jpg)
![Page 6: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/6.jpg)
If only there was an untapped reservoir of talent close by....
![Page 7: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/7.jpg)
![Page 8: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/8.jpg)
The Pipeline Solution
Recruit students to work with us by teaching a graduate level class on the underpinnings of gateways.
Embrace change.
![Page 9: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/9.jpg)
What Is a Science Gateway?A Science Gateway is a user-centric environment
for conducting online computational science
![Page 10: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/10.jpg)
(Not actual IU students)
![Page 11: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/11.jpg)
What if Mark Zuckerberg Read More Chemistry and Biology Books?
![Page 12: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/12.jpg)
![Page 13: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/13.jpg)
Goal 1: Apply basic distributed computing concepts to Science
Gateways.
![Page 14: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/14.jpg)
Goal 2: Apply new architectures, methodologies, and technologies to
Science Gateways: Microservices, DevOps
![Page 15: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/15.jpg)
Browser
Web Interface Server
Application Server
Server SDK
Client SDK
IU: Big Red 2
Resource Plugins
XSEDE: Stampede XSEDE: Comet Juelich: Jureca
HTTPS
HTTP or TCP/IP
Facebook doesn’t look like this, so why should science gateways?
![Page 16: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/16.jpg)
Goal 3: Teach open source software practices
![Page 17: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/17.jpg)
Challenges for Science Gateways• Providing a rich user experience • Defining an API for the application server• Defining the right microservices.• Implementing the components, wiring them together correctly.• Supporting multiple gateway tenants• Fault tolerance for components• State management• Continuous delivery• Security management• Supporting full scientific exploratory cycle
![Page 18: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/18.jpg)
![Page 19: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/19.jpg)
What Do We Teach?
A quick survey of course material
![Page 20: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/20.jpg)
Micro-Services
DevOpsDistributed Systems
![Page 21: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/21.jpg)
Putting It All Together: Gateways and Microservices
![Page 22: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/22.jpg)
Virtualization, Containers, Docker
![Page 23: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/23.jpg)
API and Metadata Model Design
![Page 24: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/24.jpg)
Messaging Systems: RabbitMQ, Apache Kafka
![Page 25: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/25.jpg)
Resource Management and Scheduling: Mesos, Aurora, and Torque
![Page 26: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/26.jpg)
Distributed State Management: Zookeeper, Consul
![Page 27: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/27.jpg)
Security: OAuth2 and OpenIDConnect
![Page 28: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/28.jpg)
Putting It All In Place: Continuous Integration and Deployment
![Page 29: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/29.jpg)
![Page 30: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/30.jpg)
Mission Accomplished! Students learned on their own.
![Page 31: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/31.jpg)
![Page 32: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/32.jpg)
![Page 33: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/33.jpg)
![Page 34: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/34.jpg)
![Page 35: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/35.jpg)
More Information and Next Steps
• http://courses.airavata.org• Additional material in Canvas
• Details on assignments• We are interested in taking this to a broader audience
![Page 36: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/36.jpg)
Thank you for participating!
• Please offer your feedback with our 30-second evaluation:http://sciencegateways.org/webinareval
• Join us next month (March 8) for
Building a Modern Research Data Portal with Globus – Introduction to the Globus Platform
36
![Page 37: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/37.jpg)
Science!
![Page 38: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/38.jpg)
Spring 2016 Course Overview• Students organized into teams of 3-4• 8 biweekly project milestones, each depended on the previous
assignment.• Basic sequence, Part 1:
• Learn to submit jobs on IU’s Karst Cluster, MOAB scheduler• Develop core job management application• Design an API• Design a Web front end• Capture job execution metadata
• All code must be in GitHub, must build cleanly on the grader’s laptop
![Page 39: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/39.jpg)
Spring 2016 Project Milestones, Part 2• Integrate with a continuous integration and deployment system
• Travis-CI and Amazon Code-Deploy• Apache Jenkins on Jetstream
• Break up your monolithic Web applications into microservices• Combine with CI/CD
![Page 40: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/40.jpg)
Spring 2016: What Worked, What Didn’t?• Small class, we got to know the
students pretty well.• Students managed to build basic
functioning gateways that used CI/CD automatically linked to GitHub accounts.
• Students got bogged down in job submission issues.
• Students had trouble breaking up their monolithic Web applications into microservices.
![Page 41: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/41.jpg)
Fall 2016: Begin at the End• For Fall 2016, we reverse the order.• Students learned CI/CD and microservices in the first assignments.• We moved aggressively to problems of load balancing, fault tolerance,
and containerization of services.• We de-emphasized job submission to traditional clusters
• Used Apache Mesos and Apache Aurora for this.
![Page 42: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/42.jpg)
Fall 2016: What Worked, What Didn’t• All the students were able to
keep up with the assignments.• Many were using technologies for
the first time.
• Some surprises• Some teams took it as a challenge
to use different approaches.
• Didn’t have the chance to get to know students as well as before.
• Larger class, graders
• Monthly assignments were too large.
• Biweekly worked better.
![Page 43: Creating a developer pipeline by teaching gateway technologies](https://reader035.fdocuments.net/reader035/viewer/2022062820/58ac445b1a28ab99028b5063/html5/thumbnails/43.jpg)
Spring 2017: Advanced Topics• In the Spring 2016 and Fall 2016 courses, we knew the answers
(mostly).• Now in Spring 2017 course, we tackle problems we don’t know the
answers to.• Or, if you prefer, for a given problem,
• Introductory Class: Find an answer that works• Advanced Class: Find the best answer
• How do you do that?