Harnessing the Power of Python in ArcGIS Using the Conda ...xlrd, netCDF4, requests, PyPDF, pytz...
Transcript of Harnessing the Power of Python in ArcGIS Using the Conda ...xlrd, netCDF4, requests, PyPDF, pytz...
-
HarnessingthePowerofPythoninArcGISUsingtheCondaDistributionShaunWalbridge
-
https://github.com/scw/conda-uc-2016-demo
HighQualityPDF(2MB)
https://github.com/scw/conda-uc-2016-demohttps://4326.us/esri/conda-uc/uc-2016-conda-conda-demo-full.pdf
-
Python
-
WhyPython?Accessiblefornew-comers,andthe
Extensivepackagecollection(56thousandon ),broaduser-baseStronggluelanguageusedtobindtogethermanyenvironments,bothopensourceandcommercialOpensourcewithliberallicense—dowhatyouwant
mosttaughtfirstlanguageinUSuniversites
PyPI
http://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-us-universities/fulltexthttps://pypi.python.org/pypi
-
WhyPython?Inthebox:
TheSciPyStack(NumPy,SciPy,Pandas,matplotlib,sympy)
xlrd,netCDF4,requests,PyPDF,pytz
ScientificProgrammingwiththeSciPyStack
https://4326.us/esri/scipy/#/
-
WhyPython?Beyondthebox:
IntegratingOpen-SourceStatisticalPackageswithArcGISusingPythonandR—Tomorrowat10:15am,Ballroom6D
Python:ExtendingwithOtherLibraries—Todayat4:00pm,TechTheater16
DeeperDiveintoCondainDevSummitTechSessionVideo
http://video.esri.com/watch/5072/harnessing-the-power-of-python-in-arcgis-using-the-conda-distribution
-
PackageManagementforPython
Whynotpip,wheels,virtualenvs?Don’thandletheharderproblemofsystemdependencies,consideredoutofscopebyPythonpackagers—doesitendupinsite-packages?Packagedevs:OnOSXandLinux,‘easy’togetthedeps!Useasystempackagemanager(e.g.apt,brew,yum)andtheincludedcompiler(e.g.clang,gcc).It’sstillnoteasytomakereproduciblebuilds,andwhataboutWindows?
-
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagementOnlydevelopershaveaCcompilerontheirmachineAhardproblem
-
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagementOnlydevelopershaveaCcompilerontheirmachineAhardproblem
EnterConda
-
WhyConda?
ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers
Industrystandardbuiltbypeoplewhocareaboutthisspace—ContinuumAnalytics
-
WhyConda?
Itsolvesahardproblem:
Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)
BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.
-
Conda
-
CondaCross-platform:simplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Opensource:Esriisusingit,youcanuseitinyourownprojectsforothercontexts
Whatcanitinstall?Notjustscientificpackages.Itcanhelpwith:
GUItoolkits(PyQt,TKinter)C++Libraries(Boost)IDEs(Spyder,Juptyer)
-
CondaEnvironments:CanisolateaPythonenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements—includeexplicitstateinformation,notjustthepackagename.Namesaren’tenough!AlsohandlesplatformsandJupyternotebooks
-
HowDoesitWork?Condapackagescancomefromavarietyoflocations:
Ondisk(file://)Publicrepositories(AnacondaCloud,self-hosted)Privaterepositoriesanaconda.org
https://anaconda.org/
-
CondaBasicsCommandlineinterface,fornow
CondaCheatsheet
http://conda.pydata.org/docs/_downloads/conda-cheatsheet.pdf
-
CondaBasicsDemo
-
CondaBasicsActivatingenvironments,acoupleways:
UsetheshortcutsManuallyactivatetheenvironment:
cd C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3
-
CondaBasicsTostart:
AcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem
conda --help
-
CondaBasicsOnceyou’reinanenvironmentgetdetailswithinfo:
Condainfoisthestartingpoint—ittellsyouthestateoftheenvironment.
conda info
-
CondaBasicsconda info
Current conda install:
platform : win-64 conda version : 4.0.6 conda-build version : not installed python version : 3.5.1.final.0 requests version : 2.9.1 root environment : C:\ArcGIS\bin\Python (writable) default environment : C:\ArcGIS\bin\Python\envs\arcgispro-py3 envs directories : C:\ArcGIS\bin\Python\envs package cache : C:\ArcGIS\bin\Python\pkgs channel URLs : https://conda.anaconda.org/esri/win-64/ https://conda.anaconda.org/esri/noarch/ https://repo.continuum.io/pkgs/free/win-64/ https://repo.continuum.io/pkgs/free/noarch/ config file : C:\ArcGIS\bin\Python\.condarc
-
CondaBasicsconda list
# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#arcgispro 1.3 0 esricolorama 0.3.6 py34_0 defaultsfuture 0.15.2 py34_0 defaultsmatplotlib 1.4.3 np19py34_0 defaultsmsvc_runtime 1.0.1 vc10_0 [vc10] defaultsnose 1.3.7 py34_0 defaultsnumpy 1.9.3 py34_0e [arcgispro] esriopenssl 1.0.2h vc10_0 [vc10] defaultspandas 0.17.1 np19py34_0 esripip 8.1.1 py34_1 defaultspy 1.4.31 py34_0 defaultspyparsing 2.1.1 py34_0 defaultspypdf2 1.25.1 py_0 esripytest 2.9.1 py34_0 defaultspython 3.4.4 4 defaultspython-dateutil 2.5.3 py34_0 defaultspytz 2016.4 py34_0 defaultsrequests 2.9.1 py34_0 defaultsscipy 0.16.1 np19py34_0e [arcgispro] esrisetuptools 20.7.0 py34_0 defaultssix 1.10.0 py34_0 defaultssympy 0.7.6.1 py34_0 defaultsvs2010_runtime 10.00.40219.1 0 defaults
-
CondaBasicsCreatingnewenvironments:
Afewdifferentways.Canmanuallyspecifythedependencies:
Canalsouseafilewhichincludesallthedependencies:
Thesecancontainexplcitinformationaboutchannels,toensurethatthenewenvironmentpreciselymatchestherequirements.
conda create --name my_env python=3.4 numpy flask dask
conda create --name my_env --file my_sweet_depends.txt
-
Condavs…Name Means Will
Ship?
Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto
buildandrunConda.✓
Anaconda Adistribution200+packagesbuiltwithConda
AnacondaServer
Hostthefullinfrastructureinternally
-
SkikitLearnDemoHavetweetsaboutaniOSappreleasedat#SXSWWhattopeoplethinkofit?Useanaivebayesclassifiertodeterminesentiment
-
SkikitLearnDemo# Scikit learn model based Lukas Biewald's Scikit Learn class# https://github.com/lukas/scikit-class
import arcpyimport pandas as pdfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.naive_bayes import MultinomialNB
input_csv = arcpy.GetParameterAsText(0)test_string = arcpy.GetParameterAsText(1)
df = pd.read_csv(input_csv)target = df['is_there_an_emotion_directed_at_a_brand_or_product']text = df['tweet_text']
-
SkikitLearnDemofixed_text = text[pd.notnull(text)]fixed_target = target[pd.notnull(text)]
count_vect = CountVectorizer()count_vect.fit(fixed_text)counts = count_vect.transform(fixed_text)
nb = MultinomialNB()nb.fit(counts, fixed_target)
# print out our predictionarcpy.AddMessage(nb.predict(count_vect.transform([test_string][0])))
-
DeeperDive
-
MultiplePythonsCurrently:
Platform Pythonversion
Desktop Python2.7.x(2.7.10)
Pro Python3.4.x(3.4.3)
-
MultiplePythonsUpgradecode?
Doit!Youcansupport2+3withoutthatmuchworkStillneedtochangearcpy.mappingtoarcpy.mpwhenmovingfromDesktoptoPro,butnoPythonlanguagelevelchangesneeded....
But…thiscanbecostly.Formanyorganizations,asignificantburden,evenifthelanguagechangesarerelativelysmall.MultiplePythonsis
asolutiontothis.
PythonmigrationforArcGISPro
http://pro.arcgis.com/en/pro-app/arcpy/get-started/python-migration-for-arcgis-pro.htm
-
ChallengesHavetomakesureyou’rerunningtherightPython(whathappens
whenyoutypepythonatthecommandline?)WorkingtomakethiseasyaspossibleIt’llbeeasytotellinappIsolatedinstallationfixesavarietyofissues
Requiressomeusereducationoverthe“onlyonePythononthebox”model
-
WhatDoIGetOutoftheBox?
CondacommandandaCondarootPythoninstallNewmodules(e.g.requests)CondaenvironmentwithalloftheArcGISProdependenciesasCondapackages
-
HowcanIusethis?WealreadyshipyoutheSciPystack—powerfulandoutofthebox,canusetoday(Proand10.4)Canstartusingcondatoday.Minicondaisfullystand-alone,won’taffectyourglobalPython(unlessyoutellitto)Packageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.
-
WhereCanIRunThis?
ArcGISPro1.3WillbethePythoninstall.
Future:UIforinteractionTakeadvantageofmorefeaturesIntegrationwithplatform
-
fromfutureimport*EffectivelymanagecomplexsoftwaredependencieswithConda.Thousandsofpackagesexisttoday,canintegrateitintoyour
organization’sneeds.
-
Closing
-
ThanksEsriCondaTeam:
ContinuumAnalyticsforcreatingandopensourcingConda
-
RateThisSessionPleasetakeoursurvey,findsessioninappandprovidereview