Table of Contents · 2019-08-12 · Datacenter to datacenter Replication on Kubernetes Highlights...

1.1

1.2

1.2.1

1.2.1.1

1.2.1.2

1.2.1.3

1.2.1.4

1.2.2

1.2.3

1.2.4

1.3

1.3.1

1.3.2

1.4

1.5

1.5.1

1.5.2

1.5.3

1.6

1.6.1

1.6.2

1.6.2.1

1.6.2.2

1.6.3

1.6.3.1

1.6.3.2

1.6.4

1.6.4.1

1.6.4.2

1.6.4.3

1.6.5

1.6.6

1.6.6.1

1.6.6.2

1.6.6.3

1.6.6.4

1.7

1.7.1

1.7.2

1.7.3

1.7.4

TableofContentsIntroduction

GettingStarted

Installing

Linux

MacOSX

Windows

Compiling

Authentication

AccessingtheWebInterface

ComingfromSQL

Tutorials

Kubernetes

DatacentertodatacenterReplicationonKubernetes

Highlights

Scalability

Architecture

Datamodels

Limitations

Datamodels&modeling

Concepts

Databases

WorkingwithDatabases

NotesaboutDatabases

Collections

CollectionMethods

DatabaseMethods

Documents

BasicsandTerminology

CollectionMethods

DatabaseMethods

Graphs,Vertices&Edges

NamingConventions

DatabaseNames

CollectionNames

DocumentKeys

AttributeNames

Indexing

IndexBasics

Whichindextousewhen

IndexUtilization

WorkingwithIndexes

1

1.7.4.1

1.7.4.2

1.7.4.3

1.7.4.4

1.7.4.5

1.7.4.6

1.8

1.8.1

1.8.1.1

1.8.1.2

1.8.2

1.8.2.1

1.8.3

1.8.3.1

1.8.3.2

1.8.4

1.8.5

1.9

1.9.1

1.9.2

1.9.3

1.9.4

1.9.5

1.9.6

1.9.7

1.9.7.1

1.9.7.2

1.9.7.3

1.9.7.4

1.9.8

1.9.9

1.9.9.1

1.9.9.1.1

1.9.9.1.2

1.9.9.2

1.9.9.2.1

1.9.9.2.2

1.9.10

1.9.11

1.9.12

1.9.13

1.9.14

1.9.14.1

1.9.14.2

HashIndexes

Skiplists

Persistent

FulltextIndexes

GeoIndexes

VertexCentricIndexes

Graphs

GeneralGraphs

GraphManagement

GraphFunctions

SmartGraphs

SmartGraphManagement

Traversals

UsingTraversalObjects

ExampleData

WorkingwithEdges

Pregel

FoxxMicroservices

Ataglance

Gettingstarted

Servicemanifest

Servicecontext

Configuration

Dependencies

Routers

Endpoints

Middleware

Request

Response

UsingGraphQL

Sessionsmiddleware

Sessionstorages

Collectionstorage

JWTstorage

Sessiontransports

Cookietransport

Headertransport

Servingfiles

Writingtests

CrossOrigin

Scriptsandqueuedjobs

Migrating2.xservices

Migratingfrompre-2.8

manifest.json

2

1.9.14.3

1.9.14.4

1.9.14.5

1.9.14.5.1

1.9.14.5.2

1.9.14.5.3

1.9.14.5.4

1.9.14.5.5

1.9.14.5.6

1.9.14.6

1.9.14.7

1.9.14.8

1.9.15

1.9.16

1.9.17

1.9.17.1

1.9.17.2

1.9.17.3

1.10

1.10.1

1.10.2

1.10.3

1.10.4

1.10.5

1.11

1.11.1

1.11.2

1.11.2.1

1.11.2.2

1.11.2.3

1.11.2.3.1

1.11.2.3.2

1.11.2.3.3

1.11.2.3.4

1.11.3

1.11.4

1.11.4.1

1.11.4.2

1.11.4.3

1.11.4.4

1.11.4.5

1.11.4.6

1.11.4.7

1.11.4.8

applicationContext

RepositoriesandModels

Controllers

Requestcontext

Errorhandling

Before/After/Around

Requestobject

Responseobject

DependencyInjection

Sessions

AuthandOAuth2

FoxxQueries

Legacycompatibilitymode

Usermanagement

Relatedmodules

Authentication

OAuth1.0a

OAuth2.0

Transactions

Transactioninvocation

Passingparameters

Lockingandisolation

Durability

Limitations

Deployment

Singleinstance

Cluster

Mesos,DC/OS

Generic&Docker

AdvancedTopics

StandaloneAgency

Localtestsetups

Processes

Docker

MultipleDatacenters

Kubernetes

UsingtheOperator

DeploymentResourceReference

DriverConfiguration

Authentication

Scaling

Upgrading

ArangoDBConfiguration&Secrets

Metrics

3

1.11.4.9

1.11.4.10

1.11.4.11

1.11.4.12

1.11.4.13

1.11.4.14

1.12

1.13

1.13.1

1.13.1.1

1.13.1.2

1.13.1.3

1.13.1.4

1.13.1.5

1.13.1.6

1.13.1.7

1.13.1.8

1.13.1.9

1.13.2

1.13.2.1

1.13.2.2

1.13.2.3

1.13.3

1.13.4

1.13.5

1.13.6

1.13.7

1.13.7.1

1.13.8

1.13.8.1

1.13.8.2

1.13.8.3

1.13.8.4

1.13.8.5

1.13.8.6

1.13.8.7

1.13.8.8

1.13.8.9

1.13.8.10

1.13.8.11

1.13.8.12

1.13.9

1.13.10

1.13.11

Services&Loadbalancer

DeploymentReplicationResourceReference

Storage

StorageResource

TLS

Troubleshooting

Backup&Restore

Administration

WebInterface

Dashboard

Cluster

Collections

Document

Queries

Graphs

Services

Users

Logs

ArangoDBShell

ShellOutput

Configuration

Details

Arangoimp

Arangodump

Arangorestore

Arangoexport

ManagingUsers

InArangosh

ServerConfiguration

OperatingSystemConfiguration

ManagingEndpoints

SSLConfiguration

LDAPOptions

LoggingOptions

GeneralOptions

Write-AheadLogOptions

CompactionOptions

ClusterOptions

RocksDBEngineOptions

HashCacheOptions

AsynchronousTasks

Durability

Encryption

Auditing

4

1.13.11.1

1.13.11.2

1.13.12

1.13.12.1

1.13.12.1.1

1.13.12.1.2

1.13.12.1.3

1.13.12.1.4

1.13.12.1.5

1.13.12.2

1.13.12.2.1

1.13.12.2.2

1.13.12.3

1.13.13

1.13.14

1.13.14.1

1.13.14.2

1.13.14.3

1.13.14.4

1.13.14.5

1.13.14.6

1.13.14.7

1.13.14.8

1.13.14.9

1.13.14.10

1.14

1.14.1

1.14.2

1.14.3

1.14.4

1.14.5

1.15

1.15.1

1.15.2

1.15.3

1.16

1.16.1

1.16.2

1.16.3

1.16.4

1.16.5

1.16.6

1.16.7

1.16.8

Configuration

Events

Replication

AsynchronousReplication

Components

Per-DatabaseSetup

Server-LevelSetup

SyncingCollections

ReplicationLimitations

SynchronousReplication

Implementation

Configuration

SatelliteCollections

Sharding

Upgrading

Upgradingto3.3

Upgradingto3.2

Upgradingto3.1

Upgradingto3.0

Upgradingto2.8

Upgradingto2.6

Upgradingto2.5

Upgradingto2.4

Upgradingto2.3

Upgradingto2.2

Troubleshooting

arangod

EmergencyConsole

Arangoinspect

DatafileDebugger

Arangobench

Architecture

SingleInstancevs.Cluster

Write-aheadlog

StorageEngines

Releasenotes

WhatsNewin3.3

KnownIssuesin3.3

Incompatiblechangesin3.3

WhatsNewin3.2

KnownIssuesin3.2


WhatsNewin3.1


5

1.16.9

1.16.10

1.16.11

1.16.12

1.16.13

1.16.14

1.16.15

1.16.16

1.16.17

1.16.18

1.16.19

1.16.20

1.16.21

1.16.22

1.16.23

1.16.24

1.17

1.17.1

1.17.1.1

1.17.1.2

1.17.2

1.17.2.1

1.17.2.2

1.17.2.3

1.17.2.4

1.17.2.5

1.17.2.6

1.17.2.7

1.17.2.8

1.17.2.9

1.17.3

1.17.3.1

1.17.3.1.1

1.17.3.1.2

1.17.3.1.3

1.17.3.1.4

1.17.3.1.5

1.17.3.2

1.17.3.2.1

1.17.3.2.2

1.17.3.2.3

1.17.4

1.17.5

WhatsNewin3.0


WhatsNewin2.8


WhatsNewin2.7


WhatsNewin2.6


WhatsNewin2.5


WhatsNewin2.4


WhatsNewin2.3


WhatsNewin2.2

WhatsNewin2.1

Appendix

References

db

collection

JavaScriptModules

@arangodb

console

crypto

fs

request

actions

queries

Write-aheadlog

TaskManagement

Deprecated

SimpleQueries

SequentialAccess

Pagination

ModificationQueries

GeoQueries

FulltextQueries

Actions

DeliveringHTMLPages

JsonObjects

Modifying

Errorcodesandmeanings

Glossary

6

ArangoDBv3.3.23DocumentationWelcometotheArangoDBdocumentation!

NewandeagertotryoutArangoDB?Startrightawaywithourbeginner'sguide:GettingStarted

Structure

Thedocumentationisorganizedinfourhandbooks:

ThismanualdescribesArangoDBanditsfeaturesindetailforyouasauser,developerandadministrator.TheAQLhandbookexplainsArangoDB'squerylanguageAQL.TheHTTPhandbookdescribestheinternalAPIofArangoDBthatisusedtocommunicatewithclients.Ingeneral,theHTTPhandbookwillbeofinteresttodriverdevelopers.Ifyouuseanyoftheexistingdriversforthelanguageofyourchoice,youcanskipthishandbook.Ourcookbookwithrecipesforspecificproblemsandsolutions.

Featuresareillustratedwithinteractiveusageexamples;youcancut'n'pastethemintoarangoshtotrythemout.TheHTTPREST-APIfordriverdevelopersisdemonstratedwithcut'n'pasterecipesintendedtobeusedwiththecURL.Driversmayprovidetheirownexamplesbasedonthese.jsbasedexamplestoimproveunderstandabilityfortheirrespectiveusers,i.e.forthejavadriversomeofthesamplesarere-implemented.

KeyFeatures

ArangoDBisanativemulti-model,open-sourcedatabasewithflexibledatamodelsfordocuments,graphs,andkey-values.BuildhighperformanceapplicationsusingaconvenientSQL-likequerylanguageorJavaScriptextensions.UseACIDtransactionsifyourequirethem.Scalehorizontallyandverticallywithafewmouseclicks.

Keyfeaturesinclude:

installingArangoDBonaclusterisaseasyasinstallinganapponyourmobileFlexibledatamodeling:modelyourdataascombinationofkey-valuepairs,documentsorgraphs-perfectforsocialrelationsPowerfulquerylanguage(AQL)toretrieveandmodifydataUseArangoDBasanapplicationserverandfuseyourapplicationanddatabasetogetherformaximalthroughputTransactions:runqueriesonmultipledocumentsorcollectionswithoptionaltransactionalconsistencyandisolationReplicationandSharding:setupthedatabaseinamaster-slaveconfigurationorspreadbiggerdatasetsacrossmultipleserversConfigurabledurability:lettheapplicationdecideifitneedsmoredurabilityormoreperformanceNo-nonsensestorage:ArangoDBusesallofthepowerofmodernstoragehardware,likeSSDandlargecachesJavaScriptforall:nolanguagezoo,youcanuseonelanguagefromyourbrowsertoyourback-endArangoDBcanbeeasilydeployedasafault-tolerantdistributedstatemachine,whichcanserveastheanimalbrainofdistributedappliancesItisopensource(ApacheLicense2.0)

Community

IfyouhavequestionsregardingArangoDB,Foxx,drivers,orthisdocumentationdon'thesitatetocontactuson:

GitHubforissuesandmisbehaviororpullrequestsGoogleGroupsfordiscussionsaboutArangoDBingeneralortoannounceyournewFoxxAppStackOverflowforquestionsaboutAQL,usagescenariosetc.Slack,ourcommunitychat

Whenreportingissues,pleasedescribe:

theenvironmentyourunArangoDBintheArangoDBversionyouusewhetheryou'reusingFoxxtheclientyou'reusing

Introduction

8

http://curl.haxx.sehttps://github.com/arangodb/arangodb-java-driver#learn-morehttps://github.com/arangodb/arangodb/issueshttps://www.arangodb.com/community/https://groups.google.com/forum/?hl=de#!forum/arangodbhttp://stackoverflow.com/questions/tagged/arangodbhttp://slack.arangodb.com

whichpartsofthedocumentationyou'reworkingwith(link)whatyouexpecttohappenwhatisactuallyhappening

Wewillrespondassoonaspossible.

Introduction

9

Gettingstarted

Overview

Thisbeginner'sguidewillmakeyoufamiliarwithArangoDB.Wewillcoverhowto

installandrunalocalArangoDBserverusethewebinterfacetointeractwithitstoreexampledatainthedatabasequerythedatabasetoretrievethedataagaineditandremoveexistingdata

InstallationHeadtoarangodb.com/download,selectyouroperatingsystemanddownloadArangoDB.Youmayalsofollowtheinstructionsonhowtoinstallwithapackagemanager,ifavailable.

IfyouinstalledabinarypackageunderLinux,theserverisautomaticallystarted.

IfyouinstalledArangoDBusinghomebrewunderMacOSX,starttheserverbyrunning/usr/local/sbin/arangod.

IfyouinstalledArangoDBunderWindowsasaservice,theserverisautomaticallystarted.Otherwise,runthearangod.exelocatedintheinstallationfolder'sbindirectory.YoumayhavetorunitasadministratortograntitwritepermissionstoC:\ProgramFiles.

Formorein-depthinformationonhowtoinstallArangoDB,aswellasavailablestartupparameters,installationinaclusterandsoon,seeInstalling.

ArangoDBofferstwostorageengines:MMFilesandRocksDB.Choosetheonewhichsuitsyourneedsbestintheinstallationprocessoronfirststartup.

SecuringtheinstallationThedefaultinstallationcontainsonedatabase_systemandausernamedroot.

DebianbasedpackagesandtheWindowsinstallerwillaskforapasswordduringtheinstallationprocess.Red-Hatbasedpackageswillsetarandompassword.Forallotherinstallationpackagesyouneedtoexecute

shell>arango-secure-installation

Thiswillaskforarootpasswordandsetsthispassword.

WebinterfaceTheserveritself(arangod)speaksHTTP/REST,butyoucanusethegraphicalwebinterfacetokeepitsimple.There'salsoarangosh,asynchronousshellforinteractionwiththeserver.Ifyou'readeveloper,youmightprefertheshellovertheGUI.Itdoesnotprovidefeatureslikesyntaxhighlightinghowever.

WhenyoustartusingArangoDBinyourproject,youwilllikelyuseanofficialorcommunity-madedriverwritteninthesamelanguageasyourproject.Driversimplementaprogramminginterfacethatshouldfeelnaturalforthatprogramminglanguage,anddoallthetalkingtotheserver.Therefore,youcanmostcertainlyignoretheHTTPAPIunlessyouwanttowriteadriveryourselforexplicitlywanttousetherawinterface.

GettingStarted

10

https://www.arangodb.com/download/

Togetfamiliarwiththedatabasesystemyoucanevenputdriversasideandusethewebinterface(codenameAardvark)forbasicinteraction.Thewebinterfacewillbecomeavailableshortlyafteryoustartedarangod.Youcanaccessitinyourbrowserathttp://localhost:8529-ifnot,pleaseseeTroubleshooting.

Bydefault,authenticationisenabled.Thedefaultuserisroot.Dependingontheinstallationmethodused,theinstallationprocesseitherpromptedfortherootpasswordorthedefaultrootpasswordisempty(seeabove).

Nextyouwillbeaskedwhichdatabasetouse.Everyserverinstancecomeswitha_systemdatabase.Selectthisdatabasetocontinue.

GettingStarted

11

http://localhost:8529

Youshouldthenbepresentedthedashboardwithserverstatisticslikethis:

Foramoredetaileddescriptionoftheinterface,seeWebInterface.

GettingStarted

12

Databases,collectionsanddocuments

Databasesaresetsofcollections.Collectionsstorerecords,whicharereferredtoasdocuments.CollectionsaretheequivalentoftablesinRDBMS,anddocumentscanbethoughtofasrowsinatable.Thedifferenceisthatyoudon'tdefinewhatcolumns(orratherattributes)therewillbeinadvance.Everydocumentinanycollectioncanhavearbitraryattributekeysandvalues.Documentsinasinglecollectionwilllikelyhaveasimilarstructureinpracticehowever,butthedatabasesystemitselfdoesnotimposeitandwilloperatestableandfastnomatterhowyourdatalookslike.

Readmoreinthedata-modelconceptschapter.

Fornow,youcanstickwiththedefault_systemdatabaseandusethewebinterfacetocreatecollectionsanddocuments.StartbyclickingtheCOLLECTIONSmenuentry,thentheAddCollectiontile.Giveitaname,e.g.users,leavetheothersettingsunchanged(wewantittobeadocumentcollection)andSaveit.Anewtilelabeledusersshouldshowup,whichyoucanclicktoopen.

TherewillbeNodocumentsyet.Clickthegreencirclewiththewhiteplusontheright-handsidetocreateafirstdocumentinthiscollection.Adialogwillaskyoufora_key.YoucanleavethefieldblankandclickCreatetoletthedatabasesystemassignanautomaticallygenerated(unique)key.Notethatthe_keypropertyisimmutable,whichmeansyoucannotchangeitoncethedocumentiscreated.Whatyoucanuseasdocumentkeyisdescribedinthenamingconventions.

Anautomaticallygeneratedkeycouldbe"9883"(_keyisalwaysastring!),andthedocument_idwouldbe"users/9883"inthatcase.Asidefromafewsystemattributes,thereisnothinginthisdocumentyet.Let'saddacustomattributebyclickingtheicontotheleftof(emptyobject),thenAppend.Twoinputfieldswillbecomeavailable,FIELD(attributekey)andVALUE(attributevalue).Typenameaskeyandyournameasvalue.Appendanotherattribute,nameitageandsetittoyourage.ClickSavetopersistthechanges.IfyouclickonCollection:usersatthetopontheright-handsideoftheArangoDBlogo,thedocumentbrowserwillshowthedocumentsintheuserscollectionandyouwillseethedocumentyoujustcreatedinthelist.

Queryingthedatabase

TimetoretrieveourdocumentusingAQL,ArangoDB'squerylanguage.Wecandirectlylookupthedocumentwecreatedviathe_id,buttherearealsootheroptions.ClicktheQUERIESmenuentrytobringupthequeryeditorandtypethefollowing(adjustthedocumentIDtomatchyourdocument):

RETURNDOCUMENT("users/9883")

ThenclickExecutetorunthequery.Theresultappearsbelowthequeryeditor:

[

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

}

]

Asyoucansee,theentiredocumentincludingthesystemattributesisreturned.DOCUMENT()isafunctiontoretrieveasingledocumentoralistofdocumentsofwhichyouknowthe_keysor_ids.Wereturntheresultofthefunctioncallasourqueryresult,whichisourdocumentinsideoftheresultarray(wecouldhavereturnedmorethanoneresultwithadifferentquery,butevenforasingledocumentasresult,westillgetanarrayatthetoplevel).

Thistypeofqueryiscalleddataaccessquery.Nodataiscreated,changedordeleted.Thereisanothertypeofquerycalleddatamodificationquery.Let'sinsertaseconddocumentusingamodificationquery:

INSERT{name:"KatieFoster",age:27}INTOusers

Thequeryisprettyself-explanatory:theINSERTkeywordtellsArangoDBthatwewanttoinsertsomething.Whattoinsert,adocumentwithtwoattributesinthiscase,followsnext.Thecurlybraces{}signifydocuments,orobjects.Whentalkingaboutrecordsinacollection,wecallthemdocuments.EncodedasJSON,wecallthemobjects.Objectscanalsobenested.Here'sanexample:

GettingStarted

13

{

"name":{

"first":"Katie",

"last":"Foster"

}

}

INTOisamandatorypartofeveryINSERToperationandisfollowedbythecollectionnamethatwewanttostorethedocumentin.Notethattherearenoquotemarksaroundthecollectionname.

Ifyourunabovequery,therewillbeanemptyarrayasresultbecausewedidnotspecifywhattoreturnusingaRETURNkeyword.Itisoptionalinmodificationqueries,butmandatoryindataaccessqueries.EvenwithRETURN,thereturnvaluecanstillbeanemptyarray,e.g.ifthespecifieddocumentwasnotfound.Despitetheemptyresult,theabovequerystillcreatedanewuserdocument.Youcanverifythiswiththedocumentbrowser.

Let'saddanotheruser,butreturnthenewlycreateddocumentthistime:

INSERT{name:"JamesHendrix",age:69}INTOusers

RETURNNEW

NEWisapseudo-variable,whichreferstothedocumentcreatedbyINSERT.Theresultofthequerywilllooklikethis:

[

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

Nowthatwehave3usersinourcollection,howtoretrievethemallwithasinglequery?Thefollowingdoesnotwork:




TherecanonlybeasingleRETURNstatementhereandasyntaxerrorisraisedifyoutrytoexecuteit.TheDOCUMENT()functionoffersasecondarysignaturetospecifymultipledocumenthandles,sowecoulddo:

RETURNDOCUMENT(["users/9883","users/9915","users/10074"])

Anarraywiththe_idsofall3documentsispassedtothefunction.Arraysaredenotedbysquarebrackets[]andtheirelementsareseparatedbycommas.

Butwhatifweaddmoreusers?Wewouldhavetochangethequerytoretrievethenewlyaddedusersaswell.Allwewanttosaywithourqueryis:"Foreveryuserinthecollectionusers,returntheuserdocument".WecanformulatethiswithaFORloop:

FORuserINusers

RETURNuser

Itexpressestoiterateovereverydocumentinusersandtouseuserasvariablename,whichwecanusetorefertothecurrentuserdocument.Itcouldalsobecalleddoc,uorahuacatlguacamole,thisisuptoyou.Itisadvisabletouseashortandself-descriptivenamehowever.

Theloopbodytellsthesystemtoreturnthevalueofthevariableuser,whichisasingleuserdocument.Alluserdocumentsarereturnedthisway:

[

{

"_key":"9915",

"_id":"users/9915",

GettingStarted

14

"_rev":"9915",

"age":27,

"name":"KatieFoster"

},

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

},

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

Youmayhavenoticedthattheorderofthereturneddocumentsisnotnecessarilythesameastheywereinserted.Thereisnoorderguaranteedunlessyouexplicitlysortthem.WecanaddaSORToperationveryeasily:

FORuserINusers

SORTuser._key

RETURNuser

Thisdoesstillnotreturnthedesiredresult:James(10074)isreturnedbeforeJohn(9883)andKatie(9915).Thereasonisthatthe_keyattributeisastringinArangoDB,andnotanumber.Theindividualcharactersofthestringsarecompared.1islowerthan9andtheresultistherefore"correct".Ifwewantedtousethenumericalvalueofthe_keyattributesinstead,wecouldconvertthestringtoanumberanduseitforsorting.Therearesomeimplicationshowever.Wearebetteroffsortingsomethingelse.Howabouttheage,indescendingorder?

FORuserINusers

SORTuser.ageDESC

RETURNuser

Theuserswillbereturnedinthefollowingorder:James(69),John(32),Katie(27).InsteadofDESCfordescendingorder,ASCcanbeusedforascendingorder.ASCisthedefaultthoughandcanbeomitted.

Wemightwanttolimittheresultsettoasubsetofusers,basedontheageattributeforexample.Let'sreturnusersolderthan30only:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser

ThiswillreturnJohnandJames(inthisorder).Katie'sageattributedoesnotfulfillthecriterion(greaterthan30),sheisonly27andthereforenotpartoftheresultset.Wecanmakeheragetoreturnheruserdocumentagain,usingamodificationquery:

UPDATE"9915"WITH{age:40}INusers

RETURNNEW

UPDATEallowstopartiallyeditanexistingdocument.ThereisalsoREPLACE,whichwouldremoveallattributes(exceptfor_keyand_id,whichremainthesame)andonlyaddthespecifiedones.UPDATEontheotherhandonlyreplacesthespecifiedattributesandkeepseverythingelseas-is.

TheUPDATEkeywordisfollowedbythedocumentkey(oradocument/objectwitha_keyattribute)toidentifywhattomodify.TheattributestoupdatearewrittenasobjectaftertheWITHkeyword.INdenotesinwhichcollectiontoperformthisoperationin,justlikeINTO(bothkeywordsareactuallyinterchangeablehere).ThefulldocumentwiththechangesappliedisreturnedifweusetheNEWpseudo-variable:

[

{

GettingStarted

15

"_key":"9915",

"_id":"users/9915",

"_rev":"12864",

"age":40,

"name":"KatieFoster"

}

]

IfweusedREPLACEinstead,thenameattributewouldbegone.WithUPDATE,theattributeiskept(thesamewouldapplytoadditionalattributesifwehadthem).

LetusrunourFILTERqueryagain,butonlyreturntheusernamesthistime:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser.name

Thiswillreturnthenamesofall3users:

[

"JohnSmith",

"KatieFoster",

"JamesHendrix"

]

Itiscalledaprojectionifonlyasubsetofattributesisreturned.Anotherkindofprojectionistochangethestructureoftheresults:

FORuserINusers

RETURN{userName:user.name,age:user.age}

Thequerydefinestheoutputformatforeveryuserdocument.TheusernameisreturnedasuserNameinsteadofname,theagekeepstheattributekeyinthisexample:

[

{

"userName":"JamesHendrix",

"age":69

},

{

"userName":"JohnSmith",

"age":32

},

{

"userName":"KatieFoster",

"age":40

}

]

Itisalsopossibletocomputenewvalues:

FORuserINusers

RETURNCONCAT(user.name,"'sageis",user.age)

CONCAT()isafunctionthatcanjoinelementstogethertoastring.Weuseitheretoreturnastatementforeveryuser.Asyoucansee,theresultsetdoesnotalwayshavetobeanarrayofobjects:

[

"JamesHendrix'sageis69",

"JohnSmith'sageis32",

"KatieFoster'sageis40"

]

GettingStarted

16

Nowlet'sdosomethingcrazy:foreverydocumentintheuserscollection,iterateoveralluserdocumentsagainandreturnuserpairs,e.g.JohnandKatie.Wecanusealoopinsidealoopforthistogetthecrossproduct(everypossiblecombinationofalluserrecords,3*3=9).Wedon'twantpairingslikeJohn+Johnhowever,solet'seliminatethemwithafiltercondition:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN[user1.name,user2.name]

Weget6pairings.PairslikeJames+JohnandJohn+Jamesarebasicallyredundant,butfairenough:

[

["JamesHendrix","JohnSmith"],

["JamesHendrix","KatieFoster"],

["JohnSmith","JamesHendrix"],

["JohnSmith","KatieFoster"],

["KatieFoster","JamesHendrix"],

["KatieFoster","JohnSmith"]

]

Wecouldcalculatethesumofbothagesandcomputesomethingnewthisway:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN{

pair:[user1.name,user2.name],

sumOfAges:user1.age+user2.age

}

WeintroduceanewattributesumOfAgesandaddupbothagesforthevalue:

[

{

"pair":["JamesHendrix","JohnSmith"],

"sumOfAges":101

},

{

"pair":["JamesHendrix","KatieFoster"],

"sumOfAges":109

},

{

"pair":["JohnSmith","JamesHendrix"],

"sumOfAges":101

},

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JamesHendrix"],

"sumOfAges":109

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Ifwewantedtopost-filteronthenewattributetoonlyreturnpairswithasumlessthan100,weshoulddefineavariabletotemporarilystorethesum,sothatwecanuseitinaFILTERstatementaswellasintheRETURNstatement:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

LETsumOfAges=user1.age+user2.age

FILTERsumOfAges<100

RETURN{

GettingStarted

17

pair:[user1.name,user2.name],

sumOfAges:sumOfAges

}

TheLETkeywordisfollowedbythedesignatedvariablename(sumOfAges),thenthere'sa=symbolandthevalueoranexpressiontodefinewhatvaluethevariableissupposedtohave.Were-useourexpressiontocalculatethesumhere.WethenhaveanotherFILTERtoskiptheunwantedpairingsandmakeuseofthevariablewedeclaredbefore.Wereturnaprojectionwithanarrayoftheusernamesandthecalculatedage,forwhichweusethevariableagain:

[

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Protip:whendefiningobjects,ifthedesiredattributekeyandthevariabletousefortheattributevaluearethesame,youcanuseashorthandnotation:{sumOfAges}insteadof{sumOfAges:sumOfAges}.

Finally,let'sdeleteoneoftheuserdocuments:

REMOVE"9883"INusers

ItdeletestheuserJohn(_key:"9883").Wecouldalsoremovedocumentsinaloop(samegoesforINSERT,UPDATEandREPLACE):

FORuserINusers

FILTERuser.age>=30

REMOVEuserINusers

Thequerydeletesalluserswhoseageisgreaterthanorequalto30.

HowtocontinueThereisalotmoretodiscoverinAQLandmuchmorefunctionalitythatArangoDBoffers.Continuereadingtheotherchaptersandexperimentwithatestdatabasetofosteryourknowledge.

IfyouwanttowritemoreAQLqueriesrightnow,havealookhere:

DataQueries:dataaccessandmodificationqueriesHigh-leveloperations:detaileddescriptionsofFOR,FILTERandmoreoperationsnotshowninthisintroductionFunctions:areferenceofallprovidedfunctions

ArangoDBprogramsTheArangoDBpackagecomeswiththefollowingprograms:

arangod:TheArangoDBdatabasedaemon.ThisserverprogramisintendedtorunasadaemonprocessandtoservethevariousclientsconnectiontotheserverviaTCP/HTTP.

arangosh:TheArangoDBshell.Aclientthatimplementsaread-eval-printloop(REPL)andprovidesfunctionstoaccessandadministratetheArangoDBserver.

arangoimp:AbulkimporterfortheArangoDBserver.ItsupportsJSONandCSV.

arangodump:AtooltocreatebackupsofanArangoDBdatabaseinJSONformat.

arangorestore:AtooltoloaddataofabackupbackintoanArangoDBdatabase.

GettingStarted

18

arango-dfdb:AdatafiledebuggerforArangoDB.ItisprimarilyintendedtobeusedduringdevelopmentofArangoDB.

arangobench:Abenchmarkandtesttool.Itcanbeusedforperformanceandserverfunctiontesting.

GettingStarted

19

InstallingFirstofall,downloadandinstallthecorrespondingRPMorDebianpackageorusehomebrewonMacOSX.Youcanfindpackagesforvariousoperationsystemsatourinstallsection,includinginstallersforWindows.

Howtodothatindetailisdescribedinthesubchaptersofthissection.

Onhowtosetupacluster,checkouttheDeploymentchapter.

Installing

20

https://www.arangodb.com/download

LinuxVisittheofficialArangoDBdownloadpageanddownloadthecorrectpackageforyourLinuxdistribution.Youcanfindbinarypackagesforthemostcommondistributionsthere.Followtheinstructionstouseyourfavoritepackagemanagerforthemajordistributions.AftersettinguptheArangoDBrepositoryyoucaneasilyinstallArangoDBusingyum,aptitude,urpmiorzypper.Debianbasedpackageswillaskforapasswordduringinstallation.ForanunattendedinstallationforDebian,seebelow.Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Forotherdistributionsortochangethepassword,runarango-secure-installationtosetarootpassword.Alternatively,seeCompilingifyouwanttobuildArangoDByourself.Startupthedatabaseserver.

Normally,thisisdonebyexecutingthefollowingcommand:

unix>/etc/init.d/arangodstart

Itwillstarttheserver,anddothataswellatsystemboottime.

Tostoptheserveryoucanusethefollowingcommand:

unix>/etc/init.d/arangodstop

TheexactcommandsdependonyourLinuxdistribution.Youmayrequirerootprivilegestoexecutethesecommands.

LinuxMint

PleaseusethecorrespondingUbuntuorDebianpackages.

UnattendedInstallation

Debianbasedpackagewillaskforapasswordduringinstallation.Forunattendedinstallation,youcansetthepasswordusingthedebconfhelpers.

echoarangodb3arangodb3/passwordpasswordNEWPASSWORD|debconf-set-selections

echoarangodb3arangodb3/password_againpasswordNEWPASSWORD|debconf-set-selections

Thecommandsshouldbeexecutedpriortotheinstallation.

Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Ifyouwanttoforceapassword,execute

ARANGODB_DEFAULT_ROOT_PASSWORD=NEWPASSWORDarango-secure-installation

Thecommandshouldbeexecutedaftertheinstallation.

Non-StandardInstallationIfyoucompiledArangoDBfromsourceanddidnotuseanyinstallationpackage–orusingnon-defaultlocationsand/ormultipleArangoDBinstancesonthesamehost–youmaywanttostarttheserverprocessmanually.Youcandosobyinvokingthearangodbinaryfromthecommandlineasshownbelow:

unix>/usr/local/sbin/arangod/tmp/vocbase

20ZZ-XX-YYT12:37:08Z[8145]INFOusingbuilt-inJavaScriptstartupfiles

20ZZ-XX-YYT12:37:08Z[8145]INFOArangoDB(version1.x.y)isreadyforbusiness

20ZZ-XX-YYT12:37:08Z[8145]INFOHaveFun!

Linux

21

https://www.arangodb.com/downloadhttp://www.microhowto.info/howto/perform_an_unattended_installation_of_a_debian_package.html

Tostopthedatabaseservergracefully,youcaneitherpressCTRL-CorbysendtheSIGINTsignaltotheserverprocess.Onmanysystemsthiscanbeachievedwiththefollowingcommand:

unix>kill-2`pidofarangod`

Onceyoustartedtheserver,thereshouldbearunninginstanceofarangod-theArangoDBdatabaseserver.

unix>psauxw|fgreparangod

arangodb145360.10.6530726423464s002S1:21pm0:00.18/usr/local/sbin/arangod

Ifthereisnosuchprocess,checkthelogfile/var/log/arangodb/arangod.logforerrors.Ifyouseealogmessagelike

2012-12-03T11:35:29Z[12882]ERRORDatabasedirectoryversion(1)islowerthanserverversion(1.2).

2012-12-03T11:35:29Z[12882]ERRORItseemslikeyouhaveupgradedtheArangoDBbinary.Ifthisiswhatyouwantedtodo,pleas

erestartwiththe--database.auto-upgradeoptiontoupgradethedatainthedatabasedirectory.

2012-12-03T11:35:29Z[12882]FATALDatabaseversioncheckfailed.Pleasestarttheserverwiththe--database.auto-upgradeopti

on

makesuretostarttheserveroncewiththe--database.auto-upgradeoption.

Notethatyoumayhavetoenableloggingfirst.Ifyoustarttheserverinashell,youshouldseeerrorsloggedthereaswell.

Linux

22

MacOSXThepreferredmethodforinstallingArangoDBunderMacOSXishomebrew.However,incaseyouarenotusinghomebrew,weprovideacommand-lineapporgraphicalappwhichcontainsalltheexecutables.

Homebrew

Ifyouareusinghomebrew,thenyoucaninstallthelatestreleasedstableversionofArangoDBusingbrewasfollows:

brewinstallarangodb

ThiswillinstallthecurrentstableversionofArangoDBandalldependencieswithinyourHomebrewtree.Notethattheserverwillbeinstalledas:

/usr/local/sbin/arangod

Youcanstarttheserverbyrunningthecommand/usr/local/sbin/arangod&.

Configurationfileislocatedat

/usr/local/etc/arangodb3/arangod.conf

TheArangoDBshellwillbeinstalledas:

/usr/local/bin/arangosh

YoucanuninstallArangoDBusing:

brewuninstallarangodb

However,incaseyoustartedArangoDBusingthelaunchctl,youneedtounloaditbeforeuninstallingtheserver:

launchctlunload~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

ThenremovetheLaunchAgent:

rm~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

Note:IfthelatestArangoDBVersionisnotshowninhomebrew,youalsoneedtoupdatehomebrew:

brewupdate

Knownissues

Performance-theLLVMdeliveredasofMacOSXElCapitanbuildsslowbinaries.UseGCCinstead,untilthisissuehasbeenfixedbyApple.theCommandlineargumentparsingdoesn'tacceptblanksinfilenames;theCLIversionbelowdoes.ifyouneedtochangeserverendpointwhilestartinghomebrewversion,youcaneditarangod.conffileanduncommentlinewithendpointneeded,e.g.:

[server]

endpoint=tcp://0.0.0.0:8529

MacOSX

23

http://brew.sh/

GraphicalApp

Incaseyouarenotusinghomebrew,wealsoprovideagraphicalapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDBinyourapplicationfolder.

CommandlineAppIncaseyouarenotusinghomebrew,wealsoprovideacommand-lineapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDB-CLIinyourapplicationfolder.

Startingtheapplicationwillstarttheserverandopenaterminalwindowshowingyouthelog-file.

ArangoDBserverhasbeenstarted

Thedatabasedirectoryislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/lib/arangodb'

Thelogfileislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/log/arangodb/arangod.log'

Youcanaccesstheserverusingabrowserat'http://127.0.0.1:8529/'

orstarttheArangoDBshell

'/Applications/ArangoDB-CLI.app/Contents/MacOS/arangosh'

Switchingtolog-filenow,killingthiswindowswillNOTstoptheserver.

2013-10-27T19:42:04Z[23840]INFOArangoDB(version1.4.devel[darwin])isreadyforbusiness.Havefun!

Notethatitispossibletoinstallboth,thehomebrewversionandthecommand-lineapp.Youshould,however,edittheconfigurationfilesofoneversionandchangetheportused.

MacOSX

24

https://www.arangodb.com/downloadhttps://www.arangodb.com/download

WindowsThedefaultinstallationdirectoryisC:\ProgramFiles\ArangoDB-3.x.x.Duringtheinstallationprocessyoumaychangethis.InthefollowingdescriptionwewillassumethatArangoDBhasbeeninstalledinthelocation.

Youhavetobecarefulwhenchoosinganinstallationdirectory.Youneedeitherwritepermissiontothisdirectoryoryouneedtomodifytheconfigurationfilefortheserverprocess.InthelattercasethedatabasedirectoryandtheFoxxdirectoryhavetobewritablebytheuser.

Single-andMultiuserInstallation

TherearetwomainmodesfortheinstallerofArangoDB.Theinstallerletsyouselect:

multiuserinstallation(default;adminprivilegesrequired)WillinstallArangoDBasservice.singleuserinstallationAllowtoinstallArangodbasnormaluser.Requiresmanualstartingofthedatabaseserver.

InstallationOptionsThecheckboxesallowyoutochoseweatheryouwantto:

chosecustominstallpathsdoanautomaticupgradekeepanbackupofyourdataaddexecutablestopathcreateadesktopicon

ornot.

CustomInstallPaths

Thischeckboxcontrolsifyouwillbeabletooverridethedefaultpathsfortheinstallationinsubsequentsteps.

Thedefaultinstallationpathsare:

MultiUserDefault:

Installation:C:\ProgramFiles\ArangoDB-3.x.xDataBase:C:\ProgramData\ArangoDBFoxxService:C:\ProgramData\ArangoDB-apps

SingleUserDefault:

Installation:C:\Users\\\AppData\Local\ArangoDB-3.x.xDataBase:C:\Users\\\AppData\Local\ArangoDBFoxxService:C:\Users\\\AppData\Local\ArangoDB-apps

Wearenotusingtheroamingpartoftheuser'sprofile,becausedoingsoavoidsthedatabeingsyncedtothewindowsdomaincontroller.

AutomaticUpgrade

Ifthischeckboxisselectedtheinstallerwillattempttoperformanautomaticupdate.FormoreinformationpleaseseeUpgradingfromPreviousVersion.

KeepBackup

Selectthistocreateabackupofyourdatabasedirectoryduringautomaticupgrade.Thebackupwillbecreatednexttoyourcurrentdatabasedirectorysuffixedbyatimestamp.

Windows

25

AddtoPath

Selectthistoaddthebinarydirectorytoyoursystem'spath(multiuserinstallation)oruser'spath(singleuserinstallation).

DesktopIcon

SelectifyouwanttheinstallertocreateDesktopIconsthatletyou:

accessthewebintefacestartthecommandlineclient(arangosh)startthedatabaseserver(singleuserinstallationonly)

UpgradingfromPreviousVersionIfyouareupgradingArangoDBfromanearlierversionyouneedtocopyyourolddatabasedirectorytothenewdefaultpaths.Upgradingwillkeepyourolddata,passwordandchoiceofstorageengineasitis.SwitchingtotheRocksDBstorageenginerequiresaexportandreimportofyourdata.

StartingIfyouinstalledArangoDBformultipleusers(asaservice)itisautomaticallystarted.OtherwiseyouneedtousethelinkthatwascreatedonyouDesktopifyouchosetolettheinstallercreatedesktopiconsor

theexecutablearangod.exelocatedin\bin.Thiswillusetheconfigurationfilearangod.conflocatedin\etc\arangodb,whichyoucanadjusttoyourneedsandusethedatadirectory\var\lib\arangodb.Thisistheplacewhereallyourdata(databasesandcollections)willbestoredbydefault.

Pleasechecktheoutputofthearangod.exeexecutablebeforegoingon.Iftheserverstartedsuccessfully,youshouldseealineArangoDBisreadyforbusiness.Havefun!attheendofitsoutput.

Wenowwishtocheckthattheinstallationisworkingcorrectlyandtodothiswewillbeusingtheadministrationwebinterface.Executearangod.exeifyouhavenotalreadydoneso,thenopenupyourwebbrowserandpointittothepage:

http://127.0.0.1:8529/

AdvancedStarting

Ifyouwanttoprovideourownstartscripts,youcansettheenvironmentvariableARANGODB_CONFIG_PATH.Thisvariableshouldpointtoadirectorycontainingtheconfigurationfiles.

UsingtheClient

ToconnecttoanalreadyrunningArangoDBserverinstance,thereisashellarangosh.exelocatedin\bin.Thisstartsashellwhichcanbeused–amongstotherthings–toadministerandqueryalocalorremoteArangoDBserver.

Notethatarangosh.exedoesNOTstartaseparateserver,itonlystartstheshell.Touseityoumusthaveaserverrunningsomewhere,e.g.byusingthearangod.exeexecutable.

arangosh.exeusesconfigurationfromthefilearangosh.conflocatedin\etc\arangodb\.Pleaseadjustthistoyourneedsifyouwanttousedifferentconnectionsettingsetc.

Uninstalling

TouninstalltheArangoserverapplicationyoucanusethewindowscontrolpanel(asyouwouldnormallyuninstallanapplication).Notehowever,thatanydatafilescreatedbytheArangoserverwillremainaswellasthedirectory.Tocompletetheuninstallationprocess,removethedatafilesandthedirectorymanually.

Windows

26

LimitationsforCygwin

PleasenotesomeimportantlimitationswhenrunningArangoDBunderCygwin:StartingArangoDBcanbestartedfromoutofaCygwinterminal,butpressingCTRL-Cwillforcefullykilltheserverprocesswithoutgivingitachancetohandlethekillsignal.Inthiscase,aregularservershutdownisnotpossible,whichmayleaveafileLOCKaroundintheserver'sdatadirectory.ThisfileneedstoberemovedmanuallytomakeArangoDBstartagain.Additionally,asArangoDBdoesnothaveachancetohandlethekillsignal,theservercannotforcefullyflushanydatatodiskonshutdown,leadingtopotentialdataloss.WhenstartingArangoDBfromaCygwinterminalitmightalsohappenthatnoerrorsareprintedintheterminaloutput.StartingArangoDBfromanMS-DOScommandpromptdoesnotimposetheselimitationsandisthusthepreferredmethod.

PleasenotethatArangoDBusesUTF-8asitsinternalencodingandthatthesystemconsolemustsupportaUTF-8codepage(65001)andfont.ItmaybenecessarytomanuallyswitchtheconsolefonttoafontthatsupportsUTF-8.

Windows

27

CompilingArangoDBfromscratchThefollowingsectionsdescribehowtocompileandbuildtheArangoDBfromscratch.ArangoDBwillcompileonmostLinuxandMacOSXsystems.WeassumethatyouusetheGNUC/C++compilerorclang/clang++tocompilethesource.ArangoDBhasbeentestedwiththesecompilers,butshouldbeabletocompilewithanyPosix-compliant,C++11-enabledcompiler.PleaseletusknowwhetheryousuccessfullycompileditwithanotherC/C++compiler.

Bydefault,cloningthegithubrepositorywillcheckoutdevel.ThisversioncontainsthedevelopmentversionoftheArangoDB.UsethisbranchifyouwanttomakechangestotheArangoDBsource.

OnWindowsyoufirstneedtoallowandenablesymlinksforyouruser.

PleasecheckoutthecookbookonhowtocompileArangoDB.

Compiling

28

https://github.com/git-for-windows/git/wiki/Symbolic-Links#allowing-non-administrators-to-create-symbolic-links

AuthenticationArangoDBallowstorestrictaccesstodatabasestocertainusers.Allusersofthesystemdatabaseareconsideredadministrators.Duringinstallationadefaultuserrootiscreated,whichhasaccesstoalldatabases.

Youshouldcreateadatabaseforyourapplicationtogetherwithauserthathasaccessrightstothisdatabase.SeeManagingUsers.

Usethearangoshtocreateanewdatabaseanduser.

arangosh>db._createDatabase("example");

arangosh>varusers=require("@arangodb/users");

arangosh>users.save("root@example","password");

arangosh>users.grantDatabase("root@example","example");

Youcannowconnecttothenewdatabaseusingtheuserroot@example.

shell>arangosh--server.username"root@example"--server.databaseexample

Authentication

29

AccessingtheWebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.ThewebinterfacecanbeaccessedviatheURL:

http://127.0.0.1:8529

Ifeverythingworksasexpected,youshouldseetheloginview:

FormoreinformationontheArangoDBwebinterface,seeWebInterface

AccessingtheWebInterface

30

ComingfromSQLIfyouworkedwitharelationaldatabasemanagementsystem(RDBMS)suchasMySQL,MariaDBorPostgreSQL,youwillbefamiliarwithitsquerylanguage,adialectofSQL(StructuredQueryLanguage).

ArangoDB'squerylanguageiscalledAQL.Therearesomesimilaritiesbetweenbothlanguagesdespitethedifferentdatamodelsofthedatabasesystems.ThemostnotabledifferenceisprobablytheconceptofloopsinAQL,whichmakesitfeelmorelikeaprogramminglanguage.Itsuitstheschema-lessmodelmorenaturalandmakesthequerylanguageverypowerfulwhileremainingeasytoreadandwrite.

TogetstartedwithAQL,havealookatourdetailedcomparisonofSQLandAQL.ItwillalsohelpyoutotranslateSQLqueriestoAQLwhenmigratingtoArangoDB.

BasicqueriesHowdobrowsevectorstranslatetodocumentqueries?

IntraditionalSQLyoumayeitherfetchallcolumnsofatablerowbyrow,usingSELECT*FROMtable,orselectasubsetofthecolumns.Thelistoftablecolumnstofetchiscommonlycalledcolumnlistorbrowsevector:

SELECTcolumnA,columnB,columnZFROMtable

Sincedocumentsaren'ttwo-dimensional,andneitherdoyouwanttobelimitedtoreturningtwo-dimensionallists,therequirementsforaquerylanguagearehigher.AQListhusalittlebitmorecomplexthanplainSQLatfirst,butoffersmuchmoreflexibilityinthelongrun.Itletsyouhandlearbitrarilystructureddocumentsinconvenientways,mostlyleanedonthesyntaxusedinJavaScript.

Composingthedocumentstobereturned

TheAQLRETURNstatementreturnsoneitemperdocumentitishanded.Youcanreturnthewholedocument,orjustpartsofit.GiventhatoneDocumentisadocument(retrievedlikeLEToneDocument=DOCUMENT("myusers/3456789")forinstance),itcanbereturnedas-islikethis:

RETURNoneDocument

[

{

"_id":"myusers/3456789",

"_key":"3456789",

"_rev":"14253647",

"firstName":"John",

"lastName":"Doe",

"address":{

"city":"Gotham",

"street":"RoadToNowhere1"

},

"hobbies":[

{"name":"swimming","howFavorite":10},

{"name":"biking","howFavorite":6},

{"name":"programming","howFavorite":4}

]

}

]

Returnthehobbiessub-structureonly:

RETURNoneDocument.hobbies

[

[



ComingfromSQL

31

https://arangodb.com/why-arangodb/sql-aql-comparison/


]

]

Returnthehobbiesandtheaddress:

RETURN{

hobbies:oneDocument.hobbies,

address:oneDocument.address

}

[

{

"hobbies":[




],

"address":{

"city":"Gotham",

"street":"RoadToNowhere1"

}

}

]

Returnthefirsthobbyonly:

RETURNoneDocument.hobbies[0].name

[

"swimming"

]

Returnalistofallhobbystrings:

RETURN{hobbies:oneDocument.hobbies[*].name}

[

{"hobbies":["swimming","biking","porgramming"]}

]

MorecomplexarrayandobjectmanipulationscanbedoneusingAQLfunctionsandoperators.

ComingfromSQL

32

TutorialsKubernetes:StartArangoDBonKubernetesin5minutes

Kubernetes:DC2DConKubernetes

Tutorials

33

StartArangoDBonKubernetesin5minutesStartinganArangoDBdatabase(eithersingleserverorfullblowncluster)onKubernetesinvolvesalotofresources.

TheserversneedstoruninPods,youneedSecretsforauthentication,TLScertificatesandServicestoenablecommunicationwiththedatabase.

Usekube-arangodb,theArangoDBKubernetesOperatortogreatlysimplifythisprocess.

Inthisguide,wewillexplainwhattheArangoDBKubernetesOperatoris,howtoinstallitandhowuseittodeployyourfirstArangoDBdatabaseinaKubernetescluster.

Whatis kube-arangodb

kube-arangodbisasetoftwooperatorsthatyoudeployinyourKubernetesclusterto(1)managedeploymentsoftheArangoDBdatabaseand(2)providePersistentVolumesonlocalstorageofyournodesforoptimalstorageperformance.

NotethattheoperatorthatprovidesPersistentVolumesisnotneededtorunArangoDBdeployments.YoucanalsousePersistentVolumesprovidedbyothercontrollers.

InthisguidewewillfocusontheArangoDeploymentoperator.

Installing kube-arangodb

Toinstallkube-arangodbinyourKubernetescluster,makesureyouhaveaccesstothisclusterandtherightstodeployresourcesatclusterlevel.

Fornow,anyrecentKubernetesclusterwilldo(e.g.minikube).

Thenrun(replacewiththeversionoftheoperatorthatyouwanttoinstall):

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb//manifests/crd.yaml

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb//manifests/arango-deployment.yaml

#Optional

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb//manifests/arango-storage.yaml

ThefirstcommandinstallstwoCustomResourceDefinitionsinyourKubernetescluster:

ArangoDeploymentistheresourceusedtodeployArangoDBdatabase.ArangoLocalStorageistheresourceusedtoprovisionPersistentVolumesonlocalstorage.

ThesecondcommandinstallsaDeploymentthatrunstheoperatorthatcontrolsArangoDeploymentresources.

TheoptionalthirdcommandinstallsaDeploymentthatrunstheoperatorthatprovidesPersistentVolumesonlocaldisksoftheclusternodes.Usethiswhenrunningonbare-metalorifthereisnoprovisionerforfaststorageinyourKubernetescluster.

DeployingyourfirstArangoDBdatabaseThefirstdatabasewearegoingtodeployisasingleserverdatabase.

Createafilecalledsingle-server.yamlwiththefollowingcontent.

apiVersion:"database.arangodb.com/v1alpha"

kind:"ArangoDeployment"

metadata:

name:"single-server"

spec:

mode:Single

Kubernetes

34

NowinsertthisresourceinyourKubernetesclusterusing:

kubectlapply-fsingle-server.yaml

TheArangoDeploymentoperatorinkube-arangodbwillnowinspecttheresourceyoujustdeployedandstarttheprocesstorunasingleserverdatabase.

Toinspectthecurrentstatusofyourdeployment,run:

kubectldescribeArangoDeploymentsingle-server

#orshorter

kubectldescribearangosingle-server

Toinspectthepodscreatedforthisdeployment,run:

kubectlgetpods--selector=arango_deployment=single-server

Theresultwilllooksimilartothis:

NAMEREADYSTATUSRESTARTSAGE

single-server-sngl-cjtdxrgl-fe06f01/1Running01m

OncethepodreportsthatitishasaRunningstatusandisready,yourdatabasesavailable.

Connectingtoyourdatabase

ThesingleserverdatabaseyoudeployedinthepreviouschapterisnowavailablefromwithintheKubernetesclusteraswellasoutsideit.

AccesstothedatabasefromoutsidetheKubernetesclusterisprovidedusinganexternal-accessservice.BydefaultthisserviceisoftypeLoadBalancer.IfthistypeofserviceisnotsupportedbyyourKubernetescluster,itwillbereplacedbyaserviceoftypeNodePortafteraminute.

Toseethetypeofservicethathasbeencreated,run:

kubectlgetservicesingle-server-ea

WhentheserviceisoftheLoadBalancertype,usetheIPaddresslistedintheEXTERNAL-IPcolumnwithport8529.WhentheserviceisoftheNodePorttype,usetheIPaddressofanyofthenodesofthecluster,combinewiththehigh(>30000)portlistedinthePORT(S)column.

Nowyoucanconnectyourbrowsertohttps://:/.

Yourbrowserwillshowawarningaboutanunknowncertificate.Acceptthecertificatefornow.

Thenloginusingusernamerootandanemptypassword.

IfyouwanttodeleteyoursingleserverArangoDBdatabase,justrun:

kubectldeleteArangoDeploymentsingle-server

DeployingafullblownArangoDBclusterdatabase

Thedeploymentofafullblownclusterisverysimilartodeployingasingleserverdatabase.ThedifferenceisinthemodefieldoftheArangoDeploymentspecification.

Createafilecalledcluster.yamlwiththefollowingcontent.

apiVersion:"database.arangodb.com/v1alpha"

kind:"ArangoDeployment"

Kubernetes

35

metadata:

name:"cluster"

spec:

mode:Cluster

NowinsertthisresourceinyourKubernetesclusterusing:

kubectlapply-fcluster.yaml

Thesamecommandsusedinthesingleserverdeploymentcanbeusedtoinspectyourcluster.Justusethecorrectdeploymentname(clusterinsteadofsingle-server).

Wheretogofromhere

ArangoDBKubernetesOperator

Kubernetes

36

StartArangoDBClustertoClusterSynchronizationonKubernetesThistutorialguidesyouthroughthestepsneededtoconfigureanArangoDBdatacentertodatacenterreplicationbetweentwoArangoDBclustersrunninginKubernetes.

ThisfeatureisonlyavailableintheEnterpriseEdition

Requirements

1. Thistutorialassumesthatyouhave2ArangoDBclustersrunningin2differentKubernetesclusters.2. BothKubernetesclustersareequippedwithsupportforServicesoftypeLoadBalancer.3. Youcancreate(global)DNSnamesforconfiguredServiceswithlowpropagationtimes.E.g.useCloudflare.4. Youhave4DNSnamesavailable:

OneforthedatabaseinthesourceArangoDBcluster.E.g.src-db.mycompany.comOnefortheArangoDBsyncmastersinthesourceArangoDBcluster.E.g.src-sync.mycompany.comOneforthedatabaseinthedestinationArangoDBcluster.E.g.dst-db.mycompany.comOnefortheArangoDBsyncmastersinthedestinationArangoDBcluster.E.g.dst-sync.mycompany.com

Step1:EnableDatacenterReplicationSupportonsourceArangoDBcluster

SetyourcurrentKubernetescontexttotheKubernetessourcecluster.

EdittheArangoDeploymentofthesourceArangoDBclusters.

Set:

spec.tls.altNamesto["src-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://src-sync.mycompany.com:8629"]spec.sync.externalAccess.accessPackageSecretNamesto["src-accesspackage"]

Step2:Extractaccess-packagefromsourceArangoDBclusterRun:

kubectlgetsecretsrc-accesspackage--template='{{index.data"accessPackage.yaml"}}'|\

base64-D>accessPackage.yaml

Step3:ConfiguresourceDNSnamesRun:

kubectlgetservice

FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:

-eaUsethisIPaddressforthesrc-db.mycompany.comDNSname.-syncUsethisIPaddressforthesrc-sync.mycompany.comDNSname.


37

https://www.arangodb.com/why-arangodb/arangodb-enterprise/

TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.

Step4:EnableDatacenterReplicationSupportondestinationArangoDBcluster

SetyourcurrentKubernetescontexttotheKubernetesdestinationcluster.

EdittheArangoDeploymentofthesourceArangoDBclusters.

Set:

spec.tls.altNamesto["dst-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://dst-sync.mycompany.com:8629"]

Step5:Importaccesspackageindestinationcluster

Run:

kubectlapply-faccessPackage.yaml

Note:ThisimportstwoSecrets,containingTLSinformationaboutthesourcecluster,intothedestinationcluster

Step6:ConfiguredestinationDNSnamesRun:

kubectlgetservice

FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:

-eaUsethisIPaddressforthedst-db.mycompany.comDNSname.-syncUsethisIPaddressforthedst-sync.mycompany.comDNSname.

TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.

Step7:Createan ArangoDeploymentReplicationresource

Createayamlfile(e.g.calledsrc-to-dst-repl.yaml)withthefollowingcontent:

apiVersion:"replication.database.arangodb.com/v1alpha"

kind:"ArangoDeploymentReplication"

metadata:

name:"replication-src-to-dst"

spec:

source:

masterEndpoint:["https://src-sync.mycompany.com:8629"]

auth:

keyfileSecretName:src-accesspackage-auth

tls:

caSecretName:src-accesspackage-ca

destination:

deploymentName:

Step8:WaitforDNSnamestopropagate

WaituntiltheDNSnamesconfiguredinstep3and6resolvetotheirconfiguredIPaddresses.


38

DependingonyourDNSprovidesthiscantakeafewminutesupto24hours.

Step9:Activatereplication

Run:

kubectlapply-fsrc-to-dst-repl.yaml

Replicationfromthesourceclustertothedestinationclusterwillnowbeconfigured.

CheckthestatusofthereplicationbyinspectingthestatusoftheArangoDeploymentReplicationresourceusing:

kubectldescribeArangoDeploymentReplicationreplication-src-to-dst

Assoonasthereplicationisconfigured,theAddcollectionbuttonintheCollectionspageofthewebUI(ofthedestinationcluster)willbegrayedout.


39

Highlights

Version3.3

EnterpriseEdition

DatacentertoDatacenterReplication:ReplicatetheentirestructureandcontentofanArangoDBclusterasynchronouslytoanotherclusterinadifferentdatacenterwithArangoSync.Multi-datacentersupportmeansyoucanfallbacktoareplicaofyourclusterincaseofadisasterinonedatacenter.

EncryptedBackups:ArangodumpcancreatebackupsencryptedwithasecretkeyusingAES256blockcipher.

AllEditions

Server-levelReplication:Inadditiontoper-databasereplication,thereisnowanadditionalglobalApplier.Starttheglobalreplicationontheslaveonceandallcurrentandfuturedatabaseswillbereplicatedfromthemastertotheslaveautomatically.

AsynchronousFailover:Makeasingleserverinstanceresilientwithasecondserverinstance,oneasmasterandtheotherasasynchronouslyreplicatingslave,withautomaticfailovertotheslaveifthemastergoesdown.

AlsoseeWhat'sNewin3.3.

Version3.2

RocksDBStorageEngine:YoucannowuseasmuchdatainArangoDBasyoucanfitonyourdisk.Plus,youcanenjoyperformanceboostsonwritesbyhavingonlydocument-levellocks

Pregel:WeimplementeddistributedgraphprocessingwithPregeltodiscoverhiddenpatterns,identifycommunitiesandperformin-depthanalyticsoflargegraphdatasets.

Fault-TolerantFoxx:TheFoxxmanagementinternalshavebeenrewrittenfromthegrounduptomakesuremulti-coordinatorclustersetupsalwayskeeptheirservicesinsyncandnewcoordinatorsarefullyinitializedevenwhenallexistingcoordinatorsareunavailable.

Enterprise:Workingwithsomeofourlargestcustomers,we’veaddedfurthersecurityandscalabilityfeaturestoArangoDBEnterpriselikeLDAPintegration,EncryptionatRest,andthebrandnewSatelliteCollections.


Version3.1SmartGraphs:Scalewithgraphstoaclusterandstayperformant.WithSmartGraphsyoucanusethe"smartness"ofyourapplicationlayertoshardyourgraphefficientlytoyourmachinesandlettraversalsrunlocally.

EncryptionControl:ChooseyourlevelofSSLencryption

Auditing:KeepadetailedlogofalltheimportantthingsthathappenedinArangoDB.


Version3.0

self-organizingclusterwithsynchronousreplication,master/mastersetup,sharednothingarchitecture,clustermanagementagency.

Deeplyintegrated,nativeAQLgraphtraversal

Highlights

40

VelocyPackasnewinternalbinarystorageformataswellasforintermediateAQLvalues.

PersistentindexesviaRocksDBsuitableforsortingandrangequeries.

Foxx3.0:overhauledJSframeworkfordata-centricmicroservices

SignificantlyimprovedWebInterface


Highlights

41

https://github.com/arangodb/velocypack

ScalabilityArangoDBisadistributeddatabasesupportingmultipledatamodels,andcanthusbescaledhorizontally,thatis,byusingmanyservers,typicallybasedoncommodityhardware.Thisapproachnotonlydeliversperformanceaswellascapacityimprovements,butalsoachievesresiliencebymeansofreplicationandautomaticfail-over.Furthermore,onecanbuildsystemsthatscaletheircapacitydynamicallyupanddownautomaticallyaccordingtodemand.

OnecanalsoscaleArangoDBvertically,thatis,byusingeverlargerservers.ThereisnobuiltinlimitationinArangoDB,forexample,theserverwillautomaticallyusemorethreadsifmoreCPUsarepresent.

However,scalingverticallyhasthedisadvantagethatthecostsgrowfasterthanlinearwiththesizeoftheserver,andnoneoftheresilienceanddynamicalcapabilitiescanbeachievedinthisway.

InthischapterweexplainthedistributedarchitectureofArangoDBanddiscussitsscalabilityfeaturesandlimitations:

ArangoDB'sdistributedarchitectureDifferentdatamodelsandscalabilityLimitations

Scalability

42

ArchitectureTheclusterarchitectureofArangoDBisaCPmaster/mastermodelwithnosinglepointoffailure.With"CP"wemeanthatinthepresenceofanetworkpartition,thedatabaseprefersinternalconsistencyoveravailability.With"master/master"wemeanthatclientscansendtheirrequeststoanarbitrarynode,andexperiencethesameviewonthedatabaseregardless."Nosinglepointoffailure"meansthattheclustercancontinuetoserverequests,evenifonemachinefailscompletely.

Inthisway,ArangoDBhasbeendesignedasadistributedmulti-modeldatabase.Thissectiongivesashortoutlineontheclusterarchitectureandhowtheabovefeaturesandcapabilitiesareachieved.

StructureofanArangoDBcluster

AnArangoDBclusterconsistsofanumberofArangoDBinstanceswhichtalktoeachotheroverthenetwork.Theyplaydifferentroles,whichwillbeexplainedindetailbelow.Thecurrentconfigurationoftheclusterisheldinthe"Agency",whichisahighly-availableresilientkey/valuestorebasedonanoddnumberofArangoDBinstancesrunningRaftConsensusProtocol.

ForthevariousinstancesinanArangoDBclusterthereare4distinctroles:Agents,Coordinators,PrimaryandSecondaryDBservers.Inthefollowingsectionswewillshedlightoneachofthem.NotethatthetasksforallrolesrunthesamebinaryfromthesameDockerimage.

Agents

OneormultipleAgentsformtheAgencyinanArangoDBcluster.TheAgencyisthecentralplacetostoretheconfigurationinacluster.Itperformsleaderelectionsandprovidesothersynchronizationservicesforthewholecluster.WithouttheAgencynoneoftheothercomponentscanoperate.

Whilegenerallyinvisibletotheoutsideitistheheartofthecluster.Assuch,faulttoleranceisofcourseamusthavefortheAgency.ToachievethattheAgentsareusingtheRaftConsensusAlgorithm.ThealgorithmformallyguaranteesconflictfreeconfigurationmanagementwithintheArangoDBcluster.

AtitscoretheAgencymanagesabigconfigurationtree.Itsupportstransactionalreadandwriteoperationsonthistree,andotherserverscansubscribetoHTTPcallbacksforallchangestothetree.

Coordinators

Coordinatorsshouldbeaccessiblefromtheoutside.Thesearetheonestheclientstalkto.TheywillcoordinateclustertaskslikeexecutingqueriesandrunningFoxxservices.Theyknowwherethedataisstoredandwilloptimizewheretorunusersuppliedqueriesorpartsthereof.Coordinatorsarestatelessandcanthuseasilybeshutdownandrestartedasneeded.

PrimaryDBservers

PrimaryDBserversaretheoneswherethedataisactuallyhosted.Theyhostshardsofdataandusingsynchronousreplicationaprimarymayeitherbeleaderorfollowerforashard.

Theyshouldnotbeaccessedfromtheoutsidebutindirectlythroughthecoordinators.Theymayalsoexecutequeriesinpartorasawholewhenaskedbyacoordinator.

Secondaries

SecondaryDBserversareasynchronousreplicasofprimaries.Ifoneisusingonlysynchronousreplication,onedoesnotneedsecondariesatall.Foreachprimary,therecanbeoneormoresecondaries.Sincethereplicationworksasynchronously(eventualconsistency),thereplicationdoesnotimpedetheperformanceoftheprimaries.Ontheotherhand,theirreplicaofthedatacanbeslightlyoutofdate.Thesecondariesareperfectlysuitableforbackupsastheydon'tinterferewiththenormalclusteroperation.

ClusterID

Architecture

43

https://raft.github.io/https://raft.github.io/

Everynon-AgencyArangoDBinstanceinaclusterisassignedauniqueIDduringitsstartup.UsingitsIDanodeisidentifiablethroughoutthecluster.AllclusteroperationswillcommunicateviathisID.

Sharding

UsingtherolesoutlinedaboveanArangoDBclusterisabletodistributedatainsocalledshardsacrossmultipleprimaries.Fromtheoutsidethisprocessisfullytransparentandassuchweachievethegoalsofwhatothersystemscall"master-masterreplication".InanArangoDBclusteryoutalktoanycoordinatorandwheneveryoureadorwritedataitwillautomaticallyfigureoutwherethedataisstored(read)ortobestored(write).TheinformationabouttheshardsissharedacrossthecoordinatorsusingtheAgency.

AlsoseeShardingintheAdministrationchapter.

Manysensibleconfigurations

Thisarchitectureisveryflexibleandthusallowsmanyconfigurations,whicharesuitablefordifferentusagescenarios:

1. ThedefaultconfigurationistorunexactlyonecoordinatorandoneprimaryDBserveroneachmachine.Thisachievestheclassicalmaster/mastersetup,sincethereisaperfectsymmetrybetweenthedifferentnodes,clientscanequallywelltalktoanyoneofthecoordinatorsandallexposethesameviewtothedatastore.

2. OnecandeploymorecoordinatorsthanDBservers.ThisisasensibleapproachifoneneedsalotofCPUpowerfortheFoxxservices,becausetheyrunonthecoordinators.

3. OnecandeploymoreDBserversthancoordinatorsifmoredatacapacityisneededandthequeryperformanceisthelesserbottleneck

4. Onecandeployacoordinatoroneachmachinewhereanapplicationserver(e.g.anode.jsserver)runs,andtheAgentsandDBserversonaseparatesetofmachineselsewhere.Thisavoidsanetworkhopbetweentheapplicationserverandthedatabaseandthusdecreaseslatency.Essentially,thismovessomeofthedatabasedistributionlogictothemachinewheretheclientruns.

Theseshallsufficefornow.TheimportantpieceofinformationhereisthatthecoordinatorlayercanbescaledanddeployedindependentlyfromtheDBserverlayer.

Replication

ArangoDBofferstwowaysofdatareplicationwithinacluster,synchronousandasynchronous.Inthissectionweexplainsomedetailsandhighlighttheadvantagesanddisadvantagesrespectively.

Synchronousreplicationwithautomaticfail-over

Synchronousreplicationworksonaper-shardbasis.Oneconfiguresforeachcollection,howmanycopiesofeachshardarekeptinthecluster.Atanygiventime,oneofthecopiesisdeclaredtobethe"leader"andallotherreplicasare"followers".WriteoperationsforthisshardarealwayssenttotheDBserverwhichhappenstoholdtheleadercopy,whichinturnreplicatesthechangestoallfollowersbeforetheoperationisconsideredtobedoneandreportedbacktothecoordinator.Readoperationsareallservedbytheserverholdingtheleadercopy,thisallowstoprovidesnapshotsemanticsforcomplextransactions.

IfaDBserverfailsthatholdsafollowercopyofashard,thentheleadercannolongersynchronizeitschangestothatfollower.Afterashorttimeout(3seconds),theleadergivesuponthefollower,declaresittobeoutofsync,andcontinuesservicewithoutthefollower.Whentheserverwiththefollowercopycomesback,itautomaticallyresynchronizesitsdatawiththeleaderandsynchronousreplicationisrestored.

IfaDBserverfailsthatholdsaleadercopyofashard,thentheleadercannolongerserveanyrequests.ItwillnolongersendaheartbeattotheAgency.Therefore,asupervisionprocessrunningintheRaftleaderoftheAgency,cantakethenecessaryaction(after15secondsofmissingheartbeats),namelytopromoteoneoftheserversthatholdin-syncreplicasoftheshardtoleaderforthatshard.ThisinvolvesareconfigurationintheAgencyandleadstothefactthatcoordinatorsnowcontactadifferentDBserverforrequeststothisshard.Serviceresumes.Theothersurvivingreplicasautomaticallyresynchronizetheirdatawiththenewleader.WhentheDBserverwiththeoriginalleadercopycomesback,itnoticesthatitnowholdsafollowerreplica,resynchronizesitsdatawiththenewleaderandorderisrestored.

Architecture

44

Allsharddatasynchronizationsaredoneinanincrementalway,suchthatresynchronizationsarequick.Thistechnologyallowstomoveshards(followerandleaderones)betweenDBserverswithoutserviceinterruptions.Therefore,anArangoDBclustercanmoveallthedataonaspecificDBservertootherDBserversandthenshutdownthatserverinacontrolledway.ThisallowstoscaledownanArangoDBclusterwithoutserviceinterruption,lossoffaulttoleranceordataloss.Furthermore,onecanre-balancethedistributionoftheshards,eithermanuallyorautomatically.

AlltheseoperationscanbetriggeredviaaREST/JSONAPIorviathegraphicalwebUI.Allfail-overoperationsarecompletelyhandledwithintheArangoDBcluster.

Obviously,synchronousreplicationinvolvesacertainincreasedlatencyforwriteoperations,simplybecausethereisonemorenetworkhopwithintheclusterforeveryrequest.Thereforetheusercansetthereplicationfactorto1,whichmeansthatonlyonecopyofeachshardiskept,therebyswitchingoffsynchronousreplication.Thisisasuitablesettingforlessimportantoreasilyrecoverabledataforwhichlowlatencywriteoperationsmatter.

Asynchronousreplicationwithautomaticfail-over

Asynchronousreplicationworksdifferently,inthatitisorganizedusingprimaryandsecondaryDBservers.Eachsecondaryserverreplicatesallthedataheldonaprimarybypollinginanasynchronousway.Thisprocesshasverylittleimpactontheperformanceoftheprimary.Thedisadvantageisthatthereisadelaybetweentheconfirmationofawriteoperationthatissenttotheclientandtheactualreplicationofthedata.Ifthemasterserverfailsduringthisdelay,thencommittedandconfirmeddatacanbelost.

Nevertheless,wealsoofferautomaticfail-overwiththissetup.Contrarytothesynchronouscase,herethefail-overmanagementisdonefromoutsidetheArangoDBcluster.InafutureversionwemightmovethismanagementintothesupervisionprocessintheAgency,butasofnow,themanagementisdoneviatheMesosframeworkschedulerforArangoDB(seebelow).

ThegranularityofthereplicationisawholeArangoDBinstancewithalldatathatresidesonthatinstance,whichmeansthatyouneedtwiceasmanyinstancesaswithoutasynchronousreplication.Synchronousreplicationismoreflexibleinthatrespect,youcanhavesmallerandlargerinstances,andifonefails,thedatacanberebalancedacrosstheremainingones.

MicroservicesandzeroadministationThedesignandcapabilitiesofArangoDBaregearedtowardsusageinmodernmicroservicearchitecturesofapplications.WiththeFoxxservicesitisveryeasytodeployadatacentricmicroservicewithinanArangoDBcluster.

Inaddition,onecandeploymultipleinstancesofArangoDBwithinthesameproject.Onepartoftheprojectmightneedascalabledocumentstore,anothermightneedagraphdatabase,andyetanothermightneedthefullpowerofamulti-modeldatabaseactuallymixingthevariousdatamodels.Thereareenormousefficiencybenefitstobereapedbybeingabletouseasingletechnologyforvariousrolesinaproject.

TosimplifylifeofthedevopsinsuchascenariowetryasmuchaspossibletouseazeroadministrationapproachforArangoDB.ArunningArangoDBclusterisresilientagainstfailuresandessentiallyrepairsitselfincaseoftemporaryfailures.Seethenextsectionforfurthercapabilitiesinthisdirection.

ApacheMesosintegration

Forthedistributedsetup,weusetheApacheMesosinfrastructurebydefault.ArangoDBisafullycertifiedpackageforDC/OSandcanthusbedeployedessentiallywithafewmouseclicksorasinglecommand,onceyouhaveanexistingDC/OScluster.ButevenonaplainApacheMesosclusteronecandeployArangoDBviaMarathonwithasingleAPIcallandsomeJSONconfiguration.

Theadvantageofthisapproachisthatwecannotonlyimplementtheinitialdeployment,butalsothelatermanagementofautomaticreplacementoffailedinstancesandthescalingoftheArangoDBcluster(triggeredmanuallyorevenautomatically).SinceallmanipulationsareeitherviathegraphicalwebUIorviaJSON/RESTcalls,onecanevenimplementauto-scalingveryeasily.

ADC/OSclusterisaverynaturalenvironmenttodeploymicroservicearchitectures,sinceitissoconvenienttodeployvariousservices,includingpotentiallymultipleArangoDBclusterinstanceswithinthesameDC/OScluster.Thebuilt-inservicediscoverymakesitextremelysimpletoconnectthevariousmicroservicesandMesosautomaticallytakescareofthedistributionanddeploymentofthevarioustasks.

SeetheDeploymentchapteranditssubsectionsforinstructions.

Architecture

45

ItispossibletodeployanArangoDBclusterbysimplylaunchingabunchofDockercontainerswiththerightcommandlineoptionstolinkthemup,orevenonasinglemachinestartingmultipleArangoDBprocesses.Inthatcase,synchronousreplicationwillworkwithinthedeployedArangoDBcluster,andautomaticfail-overinthesensethatthedutiesofafailedserverwillautomaticallybeassignedtoanother,survivingone.However,sincetheArangoDBclustercannotwithinitselflaunchadditionalinstances,replacementoffailednodesisnotautomaticandscalingupanddownhastobemanagedmanually.Thisiswhywedonotrecommendthissetupforproductiondeployment.

Architecture

46

DifferentdatamodelsandscalabilityInthissectionwediscussscalabilityinthecontextofthedifferentdatamodelssupportedbyArangoDB.

Key/valuepairs

Thekey/valuestoredatamodelistheeasiesttoscale.InArangoDB,thisisimplementedinthesensethatadocumentcollectionalwayshasaprimarykey_keyattributeandintheabsenceoffurthersecondaryindexesthedocumentcollectionbehaveslikeasimplekey/valuestore.

Theonlyoperationsthatarepossibleinthiscontextaresinglekeylookupsandkey/valuepairinsertionsandupdates.If_keyistheonlyshardingattributethentheshardingisdonewithrespecttotheprimarykeyandalltheseoperationsscalelinearly.Iftheshardingisdoneusingdifferentshardkeys,thenalookupofasinglekeyinvolvesaskingallshardsandthusdoesnotscalelinearly.

Documentstore

Forthedocumentstorecaseeveninthepresenceofsecondaryindexesessentiallythesameargumentsapply,sinceanindexforashardedcollectionissimplythesameasalocalindexforeachshard.Therefore,singledocumentoperationsstillscalelinearlywiththesizeofthecluster,unlessaspecialshardingconfigurationmakeslookupsorwriteoperationsmoreexpensive.

ForadeeperanalysisofthistopicseethisblogpostinwhichgoodlinearscalabilityofArangoDBforsingledocumentoperationsisdemonstrated.

ComplexqueriesandjoinsTheAQLquerylanguageallowscomplexqueries,usingmultiplecollections,secondaryindexesaswellasjoins.Inparticularwiththelatter,scalingcanbeachallenge,sinceifthedatatobejoinedresidesondifferentmachines,alotofcommunicationhastohappen.TheAQLqueryexecutionengineorganizesadatapipelineacrosstheclustertoputtogethertheresultsinthemostefficientway.Thequeryoptimizerisawareoftheclusterstructureandknowswhatdataiswhereandhowitisindexed.Therefore,itcanarriveataninformeddecisionaboutwhatpartsofthequeryoughttorunwhereinthecluster.

Nevertheless,forcertaincomplicatedjoins,therearelimitsastowhatcanbeachieved.

Graphdatabase

Graphdatabasesareparticularlygoodatqueriesongraphsthatinvolvepathsinthegraphofanaprioriunknownlength.Forexample,findingtheshortestpathbetweentwoverticesinagraph,orfindingallpathsthatmatchacertainpatternstartingatagivenvertexaresuchexamples.

However,iftheverticesandedgesalongtheoccurringpathsaredistributedacrossthecluster,thenalotofcommunicationisnecessarybetweennodes,andperformancesuffers.Toachievegoodperformanceatscale,itisthereforenecessarytogetthedistributionofthegraphdataacrosstheshardsintheclusterright.Mostofthetime,theapplicationdevelopersandusersofArangoDBknowbest,howtheirgraphsarastructured.Therefore,ArangoDBallowsuserstospecify,accordingtowhichattributesthegraphdataissharded.Ausefulfirststepisusuallytomakesurethattheedgesoriginatingatavertexresideonthesameclusternodeasthevertex.

Datamodels

47

https://mesosphere.com/blog/2015/11/30/arangodb-benchmark-dcos/

Limitations

ArangoDBhasnobuilt-inlimitationstohorizontalscalability.ThecentralresilientAgencywilleasilysustainhundredsofDBserversandcoordinators,andtheusualdatabaseoperationsworkcompletelydecentrallyanddonotrequireassistanceoftheAgency.

Likewise,thesupervisionprocessintheAgencycaneasilydealwithlotsofservers,sinceallitsactivitiesarenotperformancecritical.

Obviously,anArangoDBclusterislimitedbytheavailableresourcesofCPU,memory,diskandnetworkbandwidthandlatency.

Limitations

48

Datamodels&modelingThischapterintroducesArangoDB'scoreconceptsandcovers

itsdatamodel(ordatamodelsrespectively),theterminologyusedthroughoutthedatabasesystemandinthisdocumentation,aswellasaspectstoconsiderwhenmodelingyourdatatostrikeabalancebetweennaturaldatastructuresandgreatperformance

Youwillalsofindusageexamplesonhowtointeractwiththedatabasesystemusingarangosh,e.g.howtocreateanddropdatabases/collections,orhowtosave,update,replaceandremovedocuments.Youcandoallthisusingthewebinterfaceaswellandmaythereforeskipthesesectionsasbeginner.

Datamodels&modeling

49

Concepts

DatabaseInteraction

ArangoDBisadatabasethatservesdocumentstoclients.ThesedocumentsaretransportedusingJSONviaaTCPconnection,usingtheHTTPprotocol.ARESTAPIisprovidedtointeractwiththedatabasesystem.

ThewebinterfacethatcomeswithArangoDB,calledAardvark,providesgraphicaluserinterfacethatiseasytouse.Aninteractiveshell,calledArangosh,isalsoshipped.Inaddition,therearesocalleddriversthatmakeiteasytousethedatabasesysteminvariousenvironmentsandprogramminglanguages.AllthesetoolsusetheHTTPinterfaceoftheserverandremovethenecessitytorollownlow-levelcodeforbasiccommunicationinmostcases.

DatamodelThedocumentsyoucanstoreinArangoDBcloselyfollowtheJSONformat,althoughtheyarestoredinabinaryformatcalledVelocyPack.Adocumentcontainszeroormoreattributes,eachoftheseattributeshavingavalue.Avaluecaneitherbeanatomictype,i.e.number,string,booleanornull,oracompoundtype,i.e.anarrayorembeddeddocument/object.Arraysandsub-objectscancontainallofthesetypes,whichmeansthatarbitrarilynesteddatastructurescanberepresentedinasingledocument.

Documentsaregroupedintocollections.Acollectioncontainszeroormoredocuments.Ifyouarefamiliarwithrelationaldatabasemanagementsystems(RDBMS)thenitissafetocomparecollectionstotablesanddocumentstorows.ThedifferenceisthatinatraditionalRDBMS,youhavetodefinecolumnsbeforeyoucanstorerecordsinatable.Suchdefinitionsarealsoknownasschemas.ArangoDBisschema-less,whichmeansthatthereisnoneedtodefinewhatattributesadocumentcanhave.Everysingledocumentcanhaveacompletelydifferentstructureandstillbestoredtogetherwithotherdocumentsinasinglecollection.Inpractice,therewillbecommondenominatorsamongthedocumentsinacollection,butthedatabasesystemitselfdoesn'tforceyoutolimityourselftoacertaindatastructure.

Therearetwotypesofcollections:documentcollection(alsoreferedtoasvertexcollectionsinthecontextofgraphs)aswellasedgecollections.Edgecollectionsstoredocumentsaswell,buttheyincludetwospecialattributes,_fromand_to,whichareusedtocreaterelationsbetweendocuments.Usually,twodocuments(vertices)storedindocumentcollectionsarelinkedbyadocument(edge)storedinanedgecollection.ThisisArangoDB'sgraphdatamodel.Itfollowsthemathematicalconceptofadirected,labeledgraph,exceptthatedgesdon'tjusthavelabels,butarefull-blowndocuments.

Collectionsexistinsideofdatabases.Therecanbeoneormanydatabases.Differentdatabasesareusuallyusedformultitenantsetups,asthedatainsidethem(collections,documentsetc.)isisolatedfromoneanother.Thedefaultdatabase_systemisspecial,becauseitcannotberemoved.Databaseusersaremanagedinthisdatabase,andtheircredentialsarevalidforalldatabasesofaserverinstance.

DataRetrievalQueriesareusedtofilterdocumentsbasedoncertaincriteria,tocomputenewdata,aswellastomanipulateordeleteexistingdocuments.Queriescanbeassimpleasa"querybyexample"orascomplexas"joins"usingmanycollectionsortraversinggraphstructures.TheyarewrittenintheArangoDBQueryLanguage(AQL).

Cursorsareusedtoiterateovertheresultofqueries,sothatyougeteasilyprocessablebatchesinsteadofonebighunk.

Indexesareusedtospeedupsearches.Therearevarioustypesofindexes,suchashashindexesandgeoindexes.

Concepts

50

https://en.wikipedia.org/wiki/JSONhttps://en.wikipedia.org/wiki/Representational_state_transferhttps://arangodb.com/downloads/arangodb-drivers/https://github.com/arangodb/velocypack#readme

HandlingDatabasesThisisanintroductiontomanagingdatabasesinArangoDBfromwithinJavaScript.

WhenyouhaveanestablishedconnectiontoArangoDB,thecurrentdatabasecanbechangedexplicitlyusingthedb._useDatabase()method.Thiswillswitchtothespecifieddatabase(provideditexistsandtheusercanconnecttoit).Fromthispointon,anyfollowingactioninthesameshellorconnectionwillusethespecifieddatabase,unlessotherwisespecified.

Note:Ifthedatabaseischanged,clientdriversneedtostorethecurrentdatabasenameontheirside,too.ThisisbecauseconnectionsinArangoDBdonotcontainanystateinformation.AllstateinformationiscontainedintheHTTPrequest/responsedata.

Toconnecttoaspecificdatabaseafterarangoshhasstartedusethecommanddescribedabove.Itisalsopossibletospecifyadatabasenamewheninvokingarangosh.Forthispurpose,usethecommand-lineparameter--server.database,e.g.

>arangosh--server.databasetest

Pleasenotethatcommands,actions,scriptsorAQLqueriesshouldneveraccessmultipledatabases,eveniftheyexist.TheonlyintendedandsupportedwayinArangoDBistouseonedatabaseatatimeforacommand,anaction,ascriptoraquery.Operationsstartedinonedatabasemustnotswitchthedatabaselaterandcontinueoperatinginanother.

Databases

51


DatabaseMethods

ThefollowingmethodsareavailabletomanagedatabasesviaJavaScript.Pleasenotethatseveralofthesemethodscanbeusedfromthe_systemdatabaseonly.

Name

returnthedatabasenamedb._name()

Returnsthenameofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._name();

_system

ID

returnthedatabaseiddb._id()

Returnstheidofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._id();

1

Path

returnthepathtodatabasefilesdb._path()

Returnsthefilesystempathofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._path();

/tmp/arangosh_MHKL2m/tmp-32508-4027864002/data/databases/database-1

isSystem

returnthedatabasetypedb._isSystem()

Returnswhetherthecurrentlyuseddatabaseisthe_systemdatabase.Thesystemdatabasehassomespecialprivilegesandproperties,forexample,databasemanagementoperationssuchascreateordropcanonlybeexecutedfromwithinthisdatabase.Additionally,the_systemdatabaseitselfcannotbedropped.

UseDatabase

changethecurrentdatabasedb._useDatabase(name)

Changesthecurrentdatabasetothedatabasespecifiedbyname.Notethatthedatabasespecifiedbynamemustalreadyexist.

Changingthedatabasemightbedisallowedinsomecontexts,forexampleserver-sideactions(includingFoxx).


52

Whenperformingthiscommandfromarangosh,thecurrentcredentials(usernameandpassword)willbere-used.Thesecredentialsmightnotbevalidtoconnecttothedatabasespecifiedbyname.Additionally,thedatabaseonlybeaccessedfromcertainendpointsonly.Inthiscase,switchingthedatabasemightnotwork,andtheconnection/sessionshouldbeclosedandrestartedwithdifferentusernameandpasswordcredentialsand/orendpointdata.

ListDatabases

returnthelistofallexistingdatabasesdb._databases()

Returnsthelistofalldatabases.Thismethodcanonlybeusedfromwithinthe_systemdatabase.

CreateDatabase

createanewdatabasedb._createDatabase(name,options,users)

Createsanewdatabasewiththenamespecifiedbyname.Therearerestrictionsfordatabasenames(seeDatabaseNames).

Notethatevenifthedatabaseiscreatedsuccessfully,therewillbenochangeintothecurrentdatabasetothenewdatabase.Changingthecurrentdatabasemustexplicitlyberequestedbyusingthedb._useDatabasemethod.

Theoptionsattributecurrentlyhasnomeaningandisreservedforfutureuse.

Theoptionalusersattributecanbeusedtocreateinitialusersforthenewdatabase.Ifspecified,itmustbealistofuserobjects.Eachuserobjectcancontainthefollowingattributes:

username:theusernameasastring.Thisattributeismandatory.passwd:theuserpasswordasastring.Ifnotspecified,thenitdefaultstoanemptystring.active:abooleanflagindicatingwhethertheuseraccountshouldbeactiveornot.Thedefaultvalueistrue.extra:anoptionalJSONobjectwithextrauserinformation.ThedatacontainedinextrawillbestoredfortheuserbutnotbeinterpretedfurtherbyArangoDB.

Ifnoinitialusersarespecified,adefaultuserrootwillbecreatedwithanemptystringpassword.ThisensuresthatthenewdatabasewillbeaccessibleviaHTTPafteritiscreated.

Youcancreateusersinadatabaseifnoinitialuserisspecified.Switchintothenewdatabase(usernameandpasswordmustbeidenticaltothecurrentsession)andaddormodifyuserswiththefollowingcommands.

require("@arangodb/users").save(username,password,true);

require("@arangodb/users").update(username,password,true);

require("@arangodb/users").remove(username);

Alternatively,youcanspecifyuserdatadirectly.Forexample:

db._createDatabase("newDB",{},[{username:"newUser",passwd:"123456",active:true}])

Thosemethodscanonlybeusedfromwithinthe_systemdatabase.

DropDatabase

dropanexistingdatabasedb._dropDatabase(name)

Dropsthedatabasespecifiedbyname.Thedatabasespecifiedbynamemustexist.

Note:Droppingdatabasesisonlypossiblefromwithinthe_systemdatabase.The_systemdatabaseitselfcannotbedropped.

Databasesaredroppedasynchronously,andwillbephysicallyremovedifallclientshavedisconnectedandreferenceshavebeengarbage-collected.

Enginestatistics

retrievestatisticsrelatedtothestorageengine-rocksdbdb._engineStats()

Returnssomestatisticsrelatedtostorageengineactivity,includingfiguresaboutdatasize,cacheusage,etc.


53

Note:CurrentlythisonlyproducesusefuloutputfortheRocksDBengine.


54

NotesaboutDatabasesPleasekeepinmindthateachdatabasecontainsitsownsystemcollections,whichneedtobesetupwhenadatabaseiscreated.Thiswillmakethecreationofadatabasetakeawhile.

Replicationisconfiguredonaper-databaselevel,meaningthatanyreplicationloggingorapplyingforanewdatabasemustbeconfiguredexplicitlyafteranewdatabasehasbeencreated.

Foxxapplicationsarealsoavailableonlyinthecontextofthedatabasetheyhavebeeninstalledin.AnewdatabasewillonlyprovideaccesstothesystemapplicationsshippedwithArangoDB(thatisthewebinterfaceatthemoment)andnootherFoxxapplicationsuntiltheyareexplicitlyinstalledfortheparticulardatabase.

NotesaboutDatabases

55

JavaScriptInterfacetoCollectionsThisisanintroductiontoArangoDB'sinterfaceforcollectionsandhowtohandlecollectionsfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

Themostimportantcallisthecalltocreateanewcollection.

AddressofaCollection

AllcollectionsinArangoDBhaveauniqueidentifierandauniquename.ArangoDBinternallyusesthecollection'suniqueidentifiertolookupcollections.Thisidentifier,however,ismanagedbyArangoDBandtheuserhasnocontroloverit.Inordertoallowuserstousetheirownnames,eachcollectionalsohasauniquenamewhichisspecifiedbytheuser.Toaccessacollectionfromtheuserperspective,thecollectionnameshouldbeused,i.e.:

Collection

db._collection(collection-name)

Acollectioniscreatedbya"db._create"call.

Forexample:Assumethatthecollectionidentifieris7254820andthenameisdemo,thenthecollectioncanbeaccessedas:

db._collection("demo")

Ifnocollectionwithsuchanameexists,thennullisreturned.

Thereisashort-cutthatcanbeusedfornon-systemcollections:

Collectionname

db.collection-name

Thiscallwilleitherreturnthecollectionnameddb.collection-nameorcreateanewonewiththatnameandasetofdefaultproperties.

Note:Creatingacollectionontheflyusingdb.collection-nameisnotrecommendanddoesnotworkinarangosh.Tocreateanewcollection,pleaseuse

Create

db._create(collection-name)

Thiscallwillcreateanewcollectioncalledcollection-name.ThismethodisadatabasemethodandisdocumentedindetailatDatabaseMethods

Synchronousreplication

StartinginArangoDB3.0,thedistributedversionofferssynchronousreplication,whichmeansthatthereistheoptiontoreplicatealldataautomaticallywithintheArangoDBcluster.Thisisconfiguredforshardedcollectionsonapercollectionbasisbyspecifyinga"replicationfactor"whenthecollectioniscreated.Areplicationfactorofkmeansthataltogetherkcopiesofeachshardarekeptintheclusteronkdifferentservers,andarekeptinsync.Thatis,everywriteoperationisautomaticallyreplicatedonallcopies.

Thisisorganisedusingaleader/followermodel.Atalltimes,oneoftheserversholdingreplicasforashardis"theleader"andallothersare"followers",thisconfigurationisheldintheAgency(seeScalabilityfordetailsoftheArangoDBclusterarchitecture).Everywriteoperationissenttotheleaderbyoneofthecoordinators,andthenreplicatedtoallfollowersbeforetheoperationisreportedtohavesucceeded.Theleaderkeepsarecordofwhichfollowersarecurrentlyinsync.Incaseofnetworkproblemsorafailureofafollower,aleadercanandwilldropafollowertemporarilyafter3seconds,suchthatservicecanresume.Induecourse,thefollowerwillautomaticallyresynchronizewiththeleadertorestoreresilience.

Collections

56

Ifaleaderfails,theclusterAgencyautomaticallyinitiatesafailoverroutineafteraround15seconds,promotingoneofthefollowerstoleader.Theotherfollowers(andtheformerleader,whenitcomesback),automaticallyresynchronizewiththenewleadertorestoreresilience.Usually,thiswholefailoverprocedurecanbehandledtransparentlyforthecoordinator,suchthattheusercodedoesnotevenseeanerrormessage.

Obviously,thisfaulttolerancecomesatacostofincreasedlatency.Eachwriteoperationneedsanadditionalnetworkroundtripforthesynchronousreplicationofthefollowers,butallreplicationoperationstoallfollowershappenconcurrently.Thisis,whythedefaultreplicationfactoris1,whichmeansnoreplication.

Fordetailsonhowtoswitchonsynchronousreplicationforacollection,seethedatabasemethoddb._create(collection-name)inthesectionaboutDatabaseMethods.

Collections

57

CollectionMethods

Drop

dropsacollectioncollection.drop(options)

Dropsacollectionandallitsindexesanddata.Inordertodropasystemcollection,anoptionsobjectwithattributeisSystemsettotruemustbespecified.

Note:droppingacollectioninacluster,whichisprototypeforsharinginothercollectionsisprohibited.Inordertobeabletodropsuchacollection,alldependentcollectionsmustbedroppedfirst.

Examples

arangosh>col=db.example;

[ArangoCollection15541,"example"(typedocument,statusloaded)]

arangosh>col.drop();

arangosh>col;

[ArangoCollection15541,"example"(typedocument,statusdeleted)]

arangosh>col=db._example;

[ArangoCollection15545,"_example"(typedocument,statusloaded)]

arangosh>col.drop({isSystem:true});

arangosh>col;

[ArangoCollection15545,"_example"(typedocument,statusdeleted)]

Truncatetruncatesacollectioncollection.truncate()

Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.

Table of Contents · 2019-08-12 · Datacenter to datacenter Replication on Kubernetes Highlights...

Documents

Transcript of Table of Contents · 2019-08-12 · Datacenter to datacenter Replication on Kubernetes Highlights...