Setting up an Amazon Web Services Account
8
Setting up an Amazon Web Services Account Diana Maclean for CS448G 2011 First of all, what do we need? For this assignment, we’ll be running MapReduce workflows on Amazon’s Elastic MapReduce framework. You’ll need to do a couple of things to get set up: 1. Create an AWS account a. Sign up for S3 (this is where you’ll store your data on the cloud) & create your main and logging buckets. i. Sign up for S3Fox (optional, but recommended) b. Sign up for EC2 (the machines we’ll use to run our workflows on) c. Sign up for EMR (the packaged framework we’ll use to run our MapReduce jobs; EMR will run your workflows on EC2 for you.) 2. Create a key pair and access your account credentials 3. Redeem your (free) credit! (And check how much you have left.) 4. Familiarize yourself with S3, EMR and EC2 1: Create an AWS account Go to http://aws.amazon.com and sign up for an account, if you do not have one already. For now, please enter the required details, including payment details if required. Amazon has generously agreed to provide each student with credit for this class; more on how to redeem this later. Once you’ve created an account, you will need to sign up for the services that we will be using for the Assignment. 1a: Sign up for S3 Amazon Simple Storage Service (S3) is just a storage framework – you can upload any data here and pay (very small amounts, unless you have very large data) for storage space. You can read more about S3 here: http://aws.amazon.com/s3 Go to the AWS Management Console, and click the “S3” tab. There should be a message informing you that you need to sign up for S3. Enter your own payment details for now – we will load credit onto the account once we’re all set up.
Transcript of Setting up an Amazon Web Services Account
SettingupanAmazonWebServicesAccount
DianaMacleanforCS448G2011Firstofall,whatdoweneed?Forthisassignment,we’llberunningMapReduceworkflowsonAmazon’sElasticMapReduceframework.You’llneedtodoacoupleofthingstogetsetup:
1. CreateanAWSaccounta. SignupforS3(thisiswhereyou’llstoreyourdataonthecloud)&
createyourmainandloggingbuckets.i. SignupforS3Fox(optional,butrecommended)
b. SignupforEC2(themachineswe’llusetorunourworkflowson)c. SignupforEMR(thepackagedframeworkwe’llusetorunour
MapReducejobs;EMRwillrunyourworkflowsonEC2foryou.)2. Createakeypairandaccessyouraccountcredentials3. Redeemyour(free)credit!(Andcheckhowmuchyouhaveleft.)4. FamiliarizeyourselfwithS3,EMRandEC2
1:CreateanAWSaccountGotohttp://aws.amazon.comandsignupforanaccount,ifyoudonothaveonealready.Fornow,pleaseentertherequireddetails,includingpaymentdetailsifrequired.Amazonhasgenerouslyagreedtoprovideeachstudentwithcreditforthisclass;moreonhowtoredeemthislater.Onceyou’vecreatedanaccount,youwillneedtosignupfortheservicesthatwewillbeusingfortheAssignment.1a:SignupforS3AmazonSimpleStorageService(S3)isjustastorageframework–youcanuploadanydatahereandpay(verysmallamounts,unlessyouhaveverylargedata)forstoragespace.YoucanreadmoreaboutS3here:http://aws.amazon.com/s3GototheAWSManagementConsole,andclickthe“S3”tab.ThereshouldbeamessageinformingyouthatyouneedtosignupforS3.Enteryourownpaymentdetailsfornow–wewillloadcreditontotheaccountoncewe’reallsetup.
WeneedS3fortworeasons:1.)anEMRworkflowrequirestheinputdatatobeonS3;2.)EMRworkflowoutputisalwayssavedtoS3.Data(orobjects)inS3arestoredinwhatwecall“buckets”.Youcanthinkofbucketsasfolders.Forthisclass,we’veputthedatayou’llneedforPart1ofAssignment2inapublicbucketcalled:cs448g‐dataYou’llseehowtoreferencethisforEMRinputlateron.Inthemeanwhile,you’llneedsomebucketsofyourownto1.)storeyourEMRoutputin,and2.)storeyourlogfilesifyouwanttodebugyourEMRruns.Onceyou’reallsignedup,youshouldseeamessagepromptingyoutogetstartedbycreatingyourownbuckets.We’llcreatethelogbucketfirst.1.IntheS3console,clickon“CreateBucket”.
2.AllS3bucketshavetohaveuniquenames,socallyourloggingbucketcs448g‐lastname‐logging.Clickon“Create”(noton“SetUpLogging>>”).
3.Yournewbucketwillappearinthesidebar,withaninformativemessagetellingyouthatitisempty.
4.Nowwe’llcreateourmainbucket.Again,clickon“CreateaBucket”.Callthisonecs448g‐lastname.We’lllinkourloggingbuckettotheonewe’recreatingnow,soclickon“SetUpLogging>>”.
5.“Enablelogging”,andstarttypinginthenameofyourloggingbucket.Itshouldappearinthedropdownmenu.Selectit,and“Create”.
You’redonecreatingyourbuckets!Awesome
1ai)SignupforS3Fox(Optional,butrecommended)YoucanmanageyourbucketsthroughtheS3console,butthereisafantasticpluginforFirefoxthatmakesuploadinganddownloadingbetweenS3andyourlocalmachinereallyeasy.Ofcourse,you’llneedtobeusingFirefox.Fromhttp://www.s3fox.net,clickon“Download”.Youshouldgetapopupwindowaskingyouifyouwanttoinstalltheplugin(examplehere:http://www.s3fox.net/DownloadPage.aspx).Onceyou’veinstalled,restartFirefox.Youshouldbeabletoaccess“S3Organizer”byclickingonthe“Tools”tabintheFirefoxtaskbar,asshownbelow.
WhenyoufirstopenS3Fox,you’llneedtoenteryourAccessKeyandyourSecretKey.Youcanfindthesebyclickingon“Account”intheAWSManagementConsole(http://aws.amazon.com/account),andthenonthe“SecurityCredentials”link.ThiswilltakeyoutoapagewhereyoucanviewyourAccessKeys.
Onceyou’vecorrectlyenteredyourcredentialsinS3Fox,youshouldseeaconsolethatlookssomethinglikethefollowing(onlyyourswon’thavefilesinityet).S3FoxwillmakeitmucheasiertouploadanddownloadyourscriptsandoutputfilesoncewestartrunningourEMRjobs.
1b:SignupforEC2IntheAWSconsole,clickonthe“EC2”tab,andsignupforEC2.
Proceedasbefore;thistimeyouwillprobablybeaskedtoverifyyouridentityviaphone.Thisshouldnottakemorethanacoupleofminutestocomplete.
Onceyou’veverifiedyouridentity,youshouldgetanemailverificationstatingthatyou’rereadytouseEC2.Awesome!YoucanreadmoreaboutEC2here:http://aws.amazon.com/ec2.EC2isessentiallyabigmassofmachinesthatyoucan“rent”torunyourownworkloadson.EC2isgreatbecauseyoucancustomizeyourEC2instancesaccordingtoyourcomputerequirements(e.g.specifyRAM,softwardetc.).Wewon’tlaunchinstancesdirectly,butwhenwerunMapReducejobsthroughEMR,EMRwilluseEC2instances.1c:SignupforEMRRepeatstep1b,exceptunderthe“ElasticMapReduce”tabontheAWSManagementConsole.YoucanreadmoreaboutEMRhere:http://aws.amazon.com/elasticmapreduceInanutshell,EMRprovidesaframeworkforrunningMapReducejobsusingdatastoredonS3,andmachinesfromtheEC2service.We’llbeworkingmostlywithEMRfromhereon.2:Accessyouraccountcredentials,andcreateakeypairIfyousignedupforS3Fox,thenyoualreadyknowhowtoaccessyouraccountcredentials.Takealookatsection1aiofthistutorial,ifnot.WhenwerunjobsonEMR,you’llneedtohaveavalidpublic/privatekeypair.YoucanalsousethesekeystosshintoEC2instancemachinesthatyoulaunched,butwewon’tgothroughthathere.(Comeandfindmeifyouwanttodothis.)Tocreateyourfirstkeypair,clickonthe“EC2”tabintheAWSManagementConsole.
Ontheright,youshouldseealinkstating“0KeyPairs”.Clickonthis.
You’llbegivenanoptionto“CreateKeyPair”.Nameyourkeypairasyouwish(trytobemorecreativethanme).
Uponprovidinganameandclickingon“Create”,yourprivatekey(a.pemfile)willautomaticallydownload.Saveitinasafeplacewhereyou’llbeabletofinditagain.
Ifyouneedtoaccessyourpublickey,you’llbeabletofinditinthesameplacewhereyoufoundyouraccountcredentials.Amazonkeepsnorecordofyourprivatekey,however,soifyouloseit,you’llneedtogenerateanewset.(Note:Youwon’treallyneedtoaccessyourprivatekeyifyouusetheAWSManagementConsole,butyouwillbeaskedtonameyourkeypaireachtimeyourunanEMRjob.)3:AddCredittoyourAccountFinally,thepartyou’vebeenholdingyourbreathfor!Adding(free)credittoyouraccount!Inordertobecredited,you’llneedyouruniqueCreditCouponCode.Ifyouhavenotreceivedthisyet,e‐mailmeandIwillissueyouone.
Onceyouhaveyourcode,gotoyouraccountpage(http://aws.amazon.com/account)andclickon“PaymentMethod”.You’llseeyourbillingdetails;ifyouscrolldown,youwillalsoseeanoptiontoredeemaCreditCoupon.
ClickonRedeem/ViewAWSCredits,andenteryourcodetogetyourcredit.PleaseemailtheCS448Gcoursestaffimmediatelyifthisdoesnotwork.Unfortunately,wecanonlygiveyousomuchfreecredit,sodon’tgotoowild!Youcancheckonhowmuchcredityouhaveleftbyclickingonthe“AccountActivity”linkfromyouraccountpage.Sometimesthiscantakeawhiletoupdate,sodon’tbesurprisedifrecentchangesaren’timmediatelyapparent.4:FamiliarizeyourselfwithS3,EMRandEC2NowyoushouldbeallsetuptostartwithAssignment2.ThemorefamiliaryouarewithAWS,however,theeasieritwillbe.BrowsearoundtheAWSsite–thereareplentyofhelpfultutorialsandresourcesthere.Partyon
