Clearing Away the Clouds: Cloud Computing at …web.stanford.edu/.../Cloud-TechBriefing.pdfCloud...

STANFORD UNIVERSITY • INFORMATION TECHNOLOGY SERVICES

Clearing Away the Clouds: Cloud Computing at Stanford

Tech Briefing Turing Auditorium, 1/29/2010

Phil Reese, IT Services


1/29/10 Clearing Away the Clouds: Cloud Computing at Stanford page 1

Topics:

  A walk down memory lane, which leads to the clouds.

  A chair with four legs.

  Several cloudy examples.

  Stanford’s forecast, sunny with a likelihood of clouds.



Several Scenarios or Use Cases:

  1- Startup looking to put up an ecommerce web site, large data base application.

  2- Development group wanting to test next version of code before putting it into production.

  3- Researchers having to do large computations after a physics experiment.

  4- Multi university researchers needing to do very large computations.

  5- Many different researchers with lots of different computations trying to leverage the same computing resources.



Scenario #1:

  Startup looking to put up an ecommerce web site, large data base application. •  Forest Gump Chocolate ecommerce site (circa 1996)

•  Started with a single large x86 server hosted in the startups’ office.



Scenario #1:



•  Moved to a collocation facility to improve bandwidth availability.



Scenario #1:




•  Interface slowness during peak times.



Scenario #1:




•  Interface slowness during peak times. •  Processing slow downs under load.



Scenario #1:




•  Interface slowness during peak times. •  Processing slow downs under load. •  Final system setup before being acquired by See’s Chocolate.



Scenario #1:

  Startup looking to put up an ecommerce web site, large data base application. •  See’s Chocolate ecommerce site (circa 2001)

•  Several web servers in front of an application and data base server.



Scenario #1:



•  Running too slow, so the application and data base portions were separated to dedicated servers.



Scenario #1:




•  Still running too slow, so multiple servers at every level.



Scenario #1:




•  Still running too slow, so multiple servers at every level. •  Works but required complex networking setup.



Scenario #1:

  Startup looking to put up an ecommerce web site, large data base application. •  Wonka’s Chocolate ecommerce site (circa 2009)

•  In addition to dedicated hardware, the site now uses cloud based resources to meet peak demands at all three layers of the services stack.



Scenario #1:

  Startup looking to put up an ecommerce web site, large data base application. •  Wonka’s Chocolate ecommerce site (circa 2009)

•  In addition to dedicated hardware, the site now uses cloud based resources to meet peak demands at all three layers of the services stack.

•  Key feature of this successful site is CLUSTERING and Cloud.



Scenario #3:

  Researchers having to do large computations after a physics experiment. (circa 1997) •  Since desktop computers were pretty slow and underpowered at

that time, the researcher would look to central computing resources.



Scenario #3:



•  These would typically be very expensive and high end special purpose computers. Names like Cray, Convex, Masspar, IBM, and HP were typical.



Scenario #3:



•  These would typically be very expensive and high end special purpose computers. Names like Convex, Masspar, IBM, and HP were typical.

•  If your university didn’t have these resources, you might be able to get some time on similar machines run at a few central places, called SuperComputing Centers, across the US.



Scenario #3:





•  As these high priced systems became out of reach and Linux and better x86 systems started to be available, clusters of x86 machines began replacing the special purpose supercomputers.



Scenario #3:





•  As these high priced systems became out of reach and Linux and better x86 systems started to be available, clusters of x86 machines began replacing the special purpose supercomputers.

•  The beginning of the GRID computing concept.



Scenario #4:

  Multi university researchers needing to do very large computations. (circa 2003) •  The grid helped individual researchers get the computing resources

needed for their work. Multiple researchers needing to work on the same data found the Grid difficult to use but was the only option.



Scenario #4:



•  Lots of effort and energy put forth for national computing centers to address this issue, since it was happening all over the country.



Scenario #4:



•  Lots of effort and energy put forth for national computing centers to address this issue, since it was happening all over the country.

•  There is always reluctance to centralizing resources when it assumes that limitations will be put on the use of those resources. •  Version of Linux •  Library availability •  Different types of science needed different applications



Two over the four legs:

  Clusters of computers at Universities   Grids were formed of the clusters so that ever more

computing power could be grouped together. •  Sounds good! What were the issues?






•  Staying in sync with your own cluster and the corollary, staying in sync with the grid you belonged to.







•  What version of Linux was in most favor at the moment?



Two of the four legs:




•  What version of Linux was in most favor at the moment? •  How to coordinate the best use of the grid? The development of

the GLOBUS Tool Kit.



Scenario #2:

  Development group wanting to test next version of code before putting it into production. (circa 2007) •  At last a simple answer, just get a matching server to the production

box and test away!



Scenario #2:


box and test away! •  Oh, the boss says that will cost too much and not be used

enough to justify the cost.



Scenario #2:



enough to justify the cost. •  Borrow your neighbors system to run the tests. With a collection of

three or groups, the purchase of a spare test system makes more sense.



Scenario #2:




three or groups, the purchase of a space test system makes more sense. •  Darn, two of the four groups are on the same production refresh

cycle and need the box at the same time!



Scenario #2:




three or groups, the purchase of a space test system makes more sense. •  Darn, two of the four groups are on the same production refresh

cycle and need the box at the same time! •  Use a VIRTUAL machine on that box so each group can have their

own configuration and not interfere with the other groups.



Three of the four legs:


computing power could be grouped together.   Virtual machines allow a single machine to be used

for multiple functions. •  Better use of idle resources •  Begins to shape a framework for standardizing system OS’s and

machine configurations. •  Virtualizes the machine so that it can easily move to different

hardware architectures.



Scenario #5:

  Many different researchers with lots of different computations trying to leverage the same computing resources. •  Clusters work great for individual researchers at individual

Universities. •  Deeper look at how clusters actually operate.



Scenario #5:



•  Grids have a pretty long history but never really made the grade since they required specific versions of code and even hardware to some extent, though they got the world thinking about sharing facilities that might have some idle time.



Scenario #5:



•  Grids have a pretty long history but never really made the grade since they required specific versions of code and even hardware to some extent, though they got the world thinking about sharing facilities that might have some idle time.

•  The answer to this turns out the be the CLOUD.



Four of the four legs:



for multiple functions.   Cloud computing

•  Arguably, Cloud computing is a combination of virtual machines, clusters of computers, grids of clusters and several additional features.







•  Arguably, Cloud computing is a combination of virtual machines, clusters of computers, grids of clusters and several additional features. •  Types of clouds:

•  Private and public







•  Arguably, Cloud computing is a combination of virtual machines, clusters of computers, grids of clusters and several additional features. •  Types of clouds:

•  Private and public •  Stats and growth



Several cloudy examples.

  Fun in the clouds! •  Types of applications that are ripe for cloud use.

•  Peak loads •  Testing/prototyping •  Parallel code






•  Clouds just aren’t for computing. •  Turns out most applications require compute cycles AND

data, typically from a storage system.







data, typically from a storage system. •  Many public and private clouds include CPU and

STORAGE as the full offering.







data, typically from a storage system. •  Many public and private clouds include CPU and

STORAGE as the full offering. •  Amazon is the poster child!

•  http://aws.amazon.com/




  Creating accounts for both EC2 and S3 •  Follow the instructions… •  Open an instance and web page •  Open Cyberduck




  Creating accounts for both EC2 and S3 •  Follow the instructions… •  Open an instance and web page •  Open Cyberduck •  Discussion about instances and coordinating instances and

scheduling.





scheduling.

  There must be more interesting apps out there •  https://rightsignature.com/ •  http://www.dropbox.com/

•  Use this to sign up and get an extra 250mb •  https://www.dropbox.com/referrals/NTg0MDMyOQ

•  http://ylastic.com/





scheduling.

  There must be more interesting apps out there •  https://rightsignature.com/ •  http://www.dropbox.com/

•  Use this to sign up and get an extra 250mb •  https://www.dropbox.com/referrals/NTg0MDMyOQ

•  http://ylastic.com/

  Don’t forget Google, Hadoop, and MapReduce



Stanford’s forecast, sunny with a likelihood of clouds.

  Using AWS over a 10gig link.   Options for educational/academic use   Private versus public



Stanford’s forecast, sunny with a likelihood of clouds.

  Using AWS over a 10gig link.   Options for educational/academic use   Private versus public

  A chair with four five legs. •  Distributed computing

•  Folding at home •  Storage at home

Clearing Away the Clouds: Cloud Computing at …web.stanford.edu/.../Cloud-TechBriefing.pdfCloud...

Documents

Transcript of Clearing Away the Clouds: Cloud Computing at …web.stanford.edu/.../Cloud-TechBriefing.pdfCloud...