iRODS Tutorial

Rules and Microservices

2

iRODS Rules and Microservices

I. Introduction to Rules and Microservices

II. Simple Rules and Database Queries

III. Complex Rules and Scheduling

3

iRODS Microservices Documentation

Available from AmazonThe integrated Rule-Oriented Data System (iRODS) Micro-service

Workbookhttp://www.amazon.com/dp/1466469129

Contains example rules for running each microservice.

On-line documentation https://www.irods.org/doxygen/

4

I. Introduction to Rules and Microservices

5

Rules• See

https://www.irods.org/index.php/Changes_and_Improvements_to_the_Rule_Language_and_the_Rule_Engine

• Rules implement data policy by running (workflows of) microservices– Retention, distribution, arrangement– Authenticity, provenance, description– Integrity, replication, synchronization– Deletion, trash cans, versioning– Archiving, staging, caching– Authentication, authorization, redaction– Access, approval, IRB, audit trails, report generation– Assessment criteria, validation– Derived data product generation, format parsing

Policy is the clear statementof how data will be managedover its life cycle.

6

Microservices

• C code

• The unit of work within iRODS

• Run by rules

• Composed into workflows by rules

7

Running Rules• Rules triggered by events/policy points are contained in the

(distributed) rule base: – iRODS/server/config/reConfigs/core.re– conditional execution– first rule with satisfied condition is executed; others are skipped

• Can be run with irule - manual execution

• Delayed execution– iqstat– iqmod

8

irule – to run a rule manually

• Example rules to tweak and run are in the software distribution at iRODS/clients/icommands/test/rules3.0

• irule -F rule_file_name.r

• Some rules can only be run by admin users

9

An example – list all microservices• Download listMS.r from /snicZone/home/pracerods/rules in

iRODS

• listMS.r (lists all available microservices)ListAvailableMS { msiListEnabledMS(*KVPairs); writeKeyValPairs("stdout", *KVPairs, ": ");}INPUT nullOUTPUT ruleExecOut

• Run it with irule:irule –F listMS.r

10

Format of a RuleRule_name{

microservice1(…,*A,…,*B);microservice2(*A,…);

}INPUT *A=”first_input", *B=”second_input"OUTPUT ruleExecOut

ORRule_name(*arg) {

on(exp) { microservice1(…,*arg,*C); microservice2(*C,…);}

}INPUT nullOUTPUT ruleExecOut

*A and *B are workflow variables.

• A rule can take arguments.• A rule can be executed conditionally.• Use “null” if there are no input parameters.

ruleExecOut accesses the internal ruleExecInfo (rei) structure managed by iRODS.

11

Rules and Parameters• Literals

– constants: strings or numbers– a variable name not beginning with a special character (#,

$ or *) is taken as string input– can only be used as input parameters (not output)

• Workflow variables• Session state variables• Persistent state variables

12

Workflow Variables (*variables)• For example, in the following workflow chain:

myRule{msiDataObjOpen(*file,*FD);msiDataObjRead(*FD,10000,*BUF); writeLine(“stdout”,*BUF);

…}INPUT *file=“/compZone/home/leesa/hello”OUTPUT ruleExecOut(‘stdout’ is a structure managed by iRODS.)

• *file is an input parameter

• *FD is output from msiDataObjOpen and input to msiDataObjRead.

• *file, *FD, and *BUF are workflow variables

13

Session Variables ($variables)• contain temporary information maintained during a server session

• contain information about client-server connection, data objects, user information, resource information, etc.

• contain information that can be sent back to the client. Example: stdout, stderr

• persistent across rule executions in the same session, so can be used to pass information between rule executions

• pre-defined by iRODS, stored as a complex C-structure (the rei structure). See

https://www.irods.org/index.php/RuleExecInfo_Structure_%26_$-variables

14

Session Variables ($variables)

• $variables map to specific locations in the REI structure - mapping contained in server/config/reConfigs/core.dvm

• Example: $objPath||rei->doi->objPath $objPath||rei->doinp->objPath $dataType||rei->doi->dataType $userNameClient||rei->uoic->userName $collName||rei->coi->collName $collParentName||rei->coi->collParentName

• https://www.irods.org/index.php/Session_State_Variables

(Mappings are not necessarily unique.)

15

Persistent State Variables (#variables)

• See iRODS Primer

• “iquest” uses these field names: iquest attrs

• https://www.irods.org/index.php/Persistent_State_Information_Variables

• iRODS/lib/core/include/rodsGenQuery.h defines the attributes available via the General Query interface.

• Names begin with ‘COL_’ (column) for easy identification in the source code; mapping to variables without COL_ contained in iRODS/lib/core/include/rodsGenQueryNames.h

16

Persistent State InformationRESC_INFORESC_COMMENTRESC_CREATE_TIMERESC_MODIFY_TIMERESC_STATUSDATA_IDDATA_COLL_IDDATA_NAMEDATA_REPL_NUMDATA_VERSIONDATA_TYPE_NAMEDATA_SIZEDATA_RESC_GROUP_NAMEDATA_RESC_NAMEDATA_PATHDATA_OWNER_NAMEDATA_OWNER_ZONEDATA_REPL_STATUSDATA_STATUSDATA_CHECKSUMDATA_EXPIRYDATA_MAP_IDDATA_COMMENTSDATA_CREATE_TIMEDATA_MODIFY_TIME

DATA_ACCESS_TYPEDATA_ACCESS_NAMEDATA_TOKEN_NAMESPACEDATA_ACCESS_USER_IDDATA_ACCESS_DATA_IDCOLL_IDCOLL_NAMECOLL_PARENT_NAMECOLL_OWNER_NAMECOLL_OWNER_ZONECOLL_MAP_IDCOLL_INHERITANCECOLL_COMMENTSCOLL_CREATE_TIMECOLL_MODIFY_TIMECOLL_ACCESS_TYPECOLL_ACCESS_NAMECOLL_TOKEN_NAMESPACECOLL_ACCESS_USER_IDCOLL_ACCESS_COLL_IDMETA_DATA_ATTR_NAMEMETA_DATA_ATTR_VALUEMETA_DATA_ATTR_UNITSMETA_DATA_ATTR_IDMETA_DATA_CREATE_TIME

ZONE_IDZONE_NAMEZONE_TYPEZONE_CONNECTIONZONE_COMMENTZONE_CREATE_TIMEZONE_MODIFY_TIMEUSER_IDUSER_NAMEUSER_TYPEUSER_ZONEUSER_DNUSER_INFOUSER_COMMENTUSER_CREATE_TIMEUSER_MODIFY_TIMERESC_IDRESC_NAMERESC_ZONE_NAMERESC_TYPE_NAMERESC_CLASS_NAMERESC_LOCRESC_VAULT_PATHRESC_FREE_SPACERESC_FREE_SPACE_TIME

17

Persistent State InformationMETA_DATA_MODIFY_TIMEMETA_COLL_ATTR_NAMEMETA_COLL_ATTR_VALUEMETA_COLL_ATTR_UNITSMETA_COLL_ATTR_IDMETA_NAMESPACE_COLLMETA_NAMESPACE_DATAMETA_NAMESPACE_RESCMETA_NAMESPACE_USERMETA_RESC_ATTR_NAMEMETA_RESC_ATTR_VALUEMETA_RESC_ATTR_UNITSMETA_RESC_ATTR_IDMETA_USER_ATTR_NAMEMETA_USER_ATTR_VALUEMETA_USER_ATTR_UNITSMETA_USER_ATTR_IDRESC_GROUP_RESC_IDRESC_GROUP_NAMEUSER_GROUP_IDUSER_GROUP_NAMERULE_EXEC_IDRULE_EXEC_NAME

RULE_EXEC_REI_FILE_PATHRULE_EXEC_USER_NAMERULE_EXEC_ADDRESSRULE_EXEC_TIMERULE_EXEC_FREQUENCYRULE_EXEC_PRIORITYRULE_EXEC_ESTIMATED_EXE_TIMERULE_EXEC_NOTIFICATION_ADDRRULE_EXEC_LAST_EXE_TIMERULE_EXEC_STATUSTOKEN_NAMESPACETOKEN_IDTOKEN_NAMETOKEN_VALUETOKEN_VALUE2TOKEN_VALUE3TOKEN_COMMENTAUDIT_OBJ_IDAUDIT_USER_IDAUDIT_ACTION_IDAUDIT_COMMENTAUDIT_CREATE_TIMEAUDIT_MODIFY_TIME

SL_HOST_NAMESL_RESC_NAMESL_CPU_USEDSL_MEM_USEDSL_SWAP_USEDSL_RUNQ_LOADSL_DISK_SPACESL_NET_INPUTSL_NET_OUTPUTSL_CREATE_TIMESLD_RESC_NAMESLD_LOAD_FACTORSLD_CREATE_TIME

18

Rule Condition• Boolean expression• Examples

1. Run if msiService succeeds:rule1 { on (msiService >= 0) { ... } }

2. Run if resource is demoResc8:rule2{ on ($rescName == demoResc8) {…} }

3. Run if the pathname begins with /x/y/z:Rule3{ on ($objPath like /x/y/z/*) {…} }

• Same rule can give different actions depending on which condition is met

• Many operators==, !=, >, <, >=, <=%%, !! (and, or)expr like reg-expr , expr not like reg-expr , expr ::= string

19

Policy Enforcement Points Rule Triggers – see core.re

• These are locations within iRODS framework where an event or state (of the environment) prompts a rule to execute – An action (eg, an icommand) may involve multiple policy

enforcements points

• Policy enforcement points– Pre-action policy (eg, selection of storage location)– Execution/action policy (eg, file deletion)– Post-action policy (eg, create secondary data products)

• Actions (the triggered rules) and policy enforcement points are contained in iRODS/server/config/reConfigs/core.re

20

core.re• Take a look at iRODS/server/config/reConfigs/core.re

• Contains rules corresponding to a number of events– Create, delete, modify users– Open, create, put, copy, replicate, delete files– Create, remove collections– Create/modify metadata

• Contains some rules to set general policy– ACL policy (strict or not)– Public user policy– Default resource policy

• An empty rule corresponds to a trigger point for which no action has been defined

21

Some Policy Enforcement Points ACTIONacCreateUseracDeleteUseracGetUserbyDN acTrashPolicyacAclPolicy acSetCreateConditionsacDataDeletePolicy acRenameLocalZoneacSetRescSchemeForCreateacRescQuotaPolicyacSetMultiReplPerRescacSetNumThreadsacVacuumacSetResourceListacSetCopyNumberacVerifyChecksum acCreateUserZoneCollectionsacDeleteUserZoneCollections acPurgeFilesacRegisterData acGetIcatResults acSetPublicUserPolicyacCreateDefaultCollectionsacDeleteDefaultCollections

PRE-ACTION POLICYacPreProcForCreateUseracPreProcForDeleteUseracPreProcForModifyUser acPreProcForModifyUserGroupacChkHostAccessControl acPreProcForCollCreateacPreProcForRmCollacPreProcForModifyAVUMetadataacPreProcForModifyCollMetaacPreProcForModifyDataObjMetaacPreProcForModifyAccessControlacPreprocForDataObjOpenacPreProcForObjRenameacPreProcForCreateResourceacPreProcForDeleteResourceacPreProcForModifyResourceacPreProcForModifyResourceGroupacPreProcForCreateTokenacPreProcForDeleteTokenacNoChkFilePathPerm acPreProcForGenQueryacSetReServerNumProcacSetVaultPathPolicy

POST-ACTION POLICYacPostProcForCreateUseracPostProcForDeleteUseracPostProcForModifyUseracPostProcForModifyUserGroupacPostProcForDeleteacPostProcForCollCreate acPostProcForRmCollacPostProcForModifyAVUMetadataacPostProcForModifyCollMetaacPostProcForModifyDataObjMetaacPostProcForModifyAccessControlacPostProcForOpen acPostProcForObjRenameacPostProcForCreateResourceacPostProcForDeleteResourceacPostProcForModifyResourceacPostProcForModifyResourceGroupacPostProcForCreateTokenacPostProcForDeleteTokenacPostProcForFilePathRegacPostProcForGenQueryacPostProcForPutacPostProcForCopyacPostProcForCreate

22

Out-of-the-Box ServicesMicroservices for…• Queries on metadata catalog

• Interaction with web services

• Invocation of external applications

• Workflow constructs (loops, conditionals, exit)

• Remote and delayed execution control

23

Microservicesprint_hello_argmsiVacuummsiQuotamsiGoodFailuremsiSetResourcemsiCheckPermissionmsiCheckOwnermsiCreateUsermsiCreateCollByAdminmsiSendMailrecover_print_hellomsiCommitmsiRollbackmsiDeleteCollByAdminmsiDeleteUsermsiAddUserToGroupmsiSetDefaultRescmsiSetRescSortSchememsiSysReplDataObjmsiStageDataObjmsiSetDataObjPreferredRescmsiSetDataObjAvoidRescmsiSortDataObjmsiSysChksumDataObjmsiSetDataTypeFromExtmsiSetNoDirectRescInpmsiSetNumThreadsmsiDeleteDisallowedmsiOprDisallowed

msiDataObjCreatemsiDataObjOpenmsiDataObjClosemsiDataObjLseekmsiDataObjReadmsiDataObjWritemsiDataObjUnlinkmsiDataObjReplmsiDataObjCopymsiExtractNaraMetadatamsiSetMultiReplPerRescmsiAdmChangeCoreIRBmsiAdmShowIRBmsiAdmShowDVMmsiAdmShowFNMmsiAdmAppendToTopOfCoreIRBmsiAdmClearAppRuleStructmsiAdmAddAppRuleStructmsiGetObjTypemsiAssociateKeyValuePairsToObjmsiExtractTemplateMDFromBufmsiReadMDTemplateIntoTagStructmsiDataObjPutmsiDataObjGetmsiDataObjChksummsiDataObjPhymvmsiDataObjRenamemsiDataObjTrimmsiCollCreate

msiRmCollmsiReplCollmsiCollReplmsiPhyPathRegmsiObjStatmsiDataObjRsyncmsiFreeBuffermsiNoChkFilePathPermmsiNoTrashCanmsiSetPublicUserOprwhileExecforExecdelayExecremoteExecforEachExecmsiSleepwriteStringwriteLinewriteBytesBufwritePosIntwriteKeyValPairsmsiGetDiffTimemsiGetSystemTimemsiHumanToSystemTimemsiStrToBytesBufmsiApplyDCMetadataTemplatemsiListEnabledMSmsiSendStdoutAsEmailmsiPrintKeyValPair

msiGetValByKeymsiAddKeyValassignifExecbreakapplyAllRulesmsiExecStrCondQuerymsiExecStrCondQueryWithOptionsmsiExecGenQuerymsiMakeQuerymsiMakeGenQuerymsiGetMoreRowsmsiAddSelectFieldToGenQuerymsiAddConditionToGenQuerymsiPrintGenQueryOutToBuffermsiExecCmdmsiSetGraftPathSchememsiSetRandomSchememsiCheckHostAccessControlmsiGetIcatTimemsiGetTaggedValueFromStringmsiXmsgServerConnectmsiXmsgCreateStreammsiCreateXmsgInpmsiSendXmsgmsiRcvXmsgmsiXmsgServerDisConnectmsiString2KeyValPairmsiStrArray2StringmsiRdaToStdout

24

MicroservicesmsiRdaToDataObjmsiRdaNoResultsmsiRdaCommitmsiAW1msiRdaRollbackmsiRenameLocalZonemsiRenameCollectionmsiAclPolicymsiRemoveKeyValuePairsFromObjmsiDataObjPutWithOptionsmsiDataObjReplWithOptionsmsiDataObjChksumWithOptionsmsiDataObjGetWithOptionsmsiSetReServerNumProcmsiGetStdoutInExecCmdOutmsiGetStderrInExecCmdOutmsiAddKeyValToMspStrmsiPrintGenQueryInpmsiTarFileExtractmsiTarFileCreatemsiPhyBundleCollmsiWriteRodsLogmsiServerMonPerfmsiFlushMonStatmsiDigestMonStatmsiSplitPathmsiGetSessionVarValuemsiAutoReplicateService

msiDataObjAutoMovemsiGetContInxFromGenQueryOutmsiSetACLmsiSetRescQuotaPolicymsiPropertiesNewmsiPropertiesClearmsiPropertiesClonemsiPropertiesAddmsiPropertiesRemovemsiPropertiesGetmsiPropertiesSetmsiPropertiesExistsmsiPropertiesToStringmsiPropertiesFromStringmsiRecursiveCollCopymsiGetDataObjACLmsiGetCollectionACLmsiGetDataObjAVUsmsiGetDataObjPSmetamsiGetCollectionPSmetamsiGetDataObjAIPmsiLoadMetadataFromDataObjmsiExportRecursiveCollMetamsiCopyAVUMetadatamsiGetUserInfomsiGetUserACLmsiCreateUserAccountsFromDataObjmsiLoadUserModsFromDataObj

msiDeleteUsersFromDataObjmsiLoadACLFromDataObjmsiGetAuditTrailInfoByUserIDmsiGetAuditTrailInfoByObjectIDmsiGetAuditTrailInfoByActionIDmsiGetAuditTrailInfoByKeywordsmsiGetAuditTrailInfoByTimeStampmsiSetDataTypemsiGuessDataTypemsiMergeDataCopiesmsiIsCollmsiIsDatamsiGetCollectionContentsReportmsiGetCollectionSizemsiStructFileBundlemsiCollectionSpidermsiFlagDataObjwithAVUmsiFlagInfectedObjs

25

Microservice Modules• A module must be compiled with the code, though pluggable

modules (coming) will no longer require recompiling the whole code

• Consult the Microservice book: The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook to see which module a microservice is contained in

• Enable that module: Enabled… yes in info.txt

• Example– > irule -F rulemsiCopyAVUMetadata.r

ERROR: rcExecMyRule error. status = -1102000 NO_MICROSERVICE_FOUND_ERRLevel 0: DEBUG: execMicroService3: no micro service foundline 12, col 2 msiFlagDataObjwithAVU(*Source,*Flag,*Status);

– msiFlagDataObjwithAVU is contained in module ERA– enable module ERA– ./irodsctl istop– ./irodssetup

26

Creating New MicroservicesModules

Define FunctionSiganture

RegistermServiceMapping

CreateInternal Function

Important!!Implement recovery mService

DescribemService

Any function can be converted into a microservice,but it’s important to implement recovery microservices

This procedure will change with coming releases.

27

II. Simple Rules and Database Queries

28

Example Rules• See iRODS/clients/icommands/test/rules3.0 in the iRODS

distribution

• See /compZone/home/leesa/rules in compZone data grid

• See /compZone/home/rods/rules in compZone data grid

• See /snicZone/home/pracerods/rules in snicZone

• See http://www.renci.org/~leesa/rules/

29

Listing the Rule Base

showCore.r

showCoreRules {# Listing of the core.re file## Input parameters:# nonemsiAdmShowCoreRE();}INPUT nullOUTPUT ruleExecOut

An admin user can execute the rule to show the rule base:–irule –vF showCore.r

30

rulemsiExecCmd.r

Run an executable script

myTestRule {#Input parameters are:# Command to be executed - located in directory irods/server/bin/cmd# Optional command argument string# Optional host address for command execution# Optional hint for remote data object path, command is executed on host# where the file is stored# Optional flag. If > 0, use the resolved physical data object path as first argument#Output parameter is:# Structure holding status, stdout, and stderr from command execution#Output:# Command result is: “Hello world written from irods”# msiExecCmd(*Cmd,*Arg,"null","null","null",*Result); msiGetStdoutInExecCmdOut(*Result,*Out); writeLine("stdout","Command result is"); writeLine("stdout","*Out");}INPUT *Cmd=”hello", *Arg="written"OUTPUT ruleExecOut

“hello” is an executable script in iRODS/server/bin/cmd.

31

Example Policy ImplementationUsing “asPostProcForPut” to implement policy, depending on resource

Data coming in to a target iRODS resource triggers a script that takes some desired action, triggers message to admin (unix) user

acPostProcForPut{ on($rescName like ”uncResc") { writeLine("serverLog","USER, OBJPATH, and FILEPATH:

$userNameClient, $objPath and $filePath"); msiExecCmd("resource-trigger.sh","$rescName $objPath

$userNameClient","null","null","null",*Out); msiSendMail("leesa@renci.org","resource $rescName","User

$userNameClient just ingested file $objPath into $rescName."); }}

• acPostProcForPut is contained in iRODS/server/config/reConfigs/core.re

• resource-trigger.sh is contained in /server/bin/cmd and must be executable!!

32

Example script resource-trigger.sh

• > more resource-trigger.sh #!/bin/sh# echo "execCmdRule: "$execCmdRule

rescName=$1objPath=$2userNameClient=$3

echo "User $userNameClient just ingested file $objPath into $rescName"

echo "User $userNameClient just ingested file $objPath into $rescName" > /tmp/resource.out

33

Example Policy ImplementationUsing asPostProcForPut to implement policy: inputs to a specific resource

Data coming in to a target iRODS collection triggers a script that takes some desired action (sending data to a remote ftp site)

acPostProcForPut{ on($objPath like ”/snicZone/home/prace-class/*") { writeLine("serverLog",

"$userNameClient sending $objPath to NCDC."); msiSplitPath($filePath,*fileDir,*fileName); msiExecCmd(“outgoing.sh","*fileDir

*fileName","null","null","null",*Out); msiSendMail("leesa@renci.org",

"send to NCDC","User $userNameClient sent $objPath to NCDC.");}

} • acPostProcForPut is contained in iRODS/server/config/reConfigs/core.re

• acPostProcForPut is the same rule in both examples! Just using different conditions.

34

Example script outgoing.sh

• > more outgoing.sh #!/bin/shHOST=ftp.****.****.***USER=anonymousPASSWD=leesa@renci.org srcDir=$1srcFile=$2echo $srcDirecho $srcFile#echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile"echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile"

> /tmp/test.out

35

iquest – for constrained querying of the iCAT • “iquest attrs” shows the attributes you can query on

• Query iCAT of remote zoneA: iquest –z zoneA …Example: User rods#compZone, logged into compZone, gives the command> iquest -z snicZone "SELECT COLL_NAME, DATA_NAME WHERE COLL_NAME like

'/snicZone/home/rods%'"Zone is snicZoneCOLL_NAME = /snicZone/home/rods#compZoneDATA_NAME = hello

------------------------------------------------------------The response comes from the snicZone iCAT, not the user’s home compZone

iCAT.

36

More on iquest

Set SQL logging to see actual SQL queries generated using iquest– Edit scripts/perl/irodsctl.pl - uncoment the line

$spLogSql = "1”;– ./irodsctl irestart– Logged into iRODS/server/log files

37

Specific QueriesMore on iquest

• iadmin asq – add SQL queryExample: show all data files with metadata, along with the metadata> iadmin asq 'select object_id, data_id, data_name, meta_attr_name, meta_attr_value

from r_objt_metamap, r_meta_main, r_data_main where r_meta_main.meta_id = r_objt_metamap.meta_id and object_id = data_id;' file-metadata

Now any user can make this queryusing iquest:> iquest –-sql file-metadata

Data grid administrators: beware what you enable with specific queries; the iCAT contains passwords, ACLs, etc. Control carefully the access you give.

Alias to the query

38

GenQuery• The primary query interface in IRODS

• msiMakeGenQuery constructs a SQL string that’s passed to the iCAT by a subsequent call to msiExecGenQuery

• Sets up a data structure, so use msiGetValByKey to get values out of the call to msiExecGenQuery

• Many iRODS commands (eg, ils) use genQuery behind the scenes

• Generates SQL to perform the query subject to access control

• Simplified SQL-like interface – user doesn’t need to know the iCAT schema to make queries

• Queries on same attributes as iquest

39

GenQuery Exampletest0.r – list all data objects with the name “hello”

mygenQuery{# Input:# string containing attribute list to select on# string containing condition# Output:# GenQInp: an internal data structure#msiMakeGenQuery("COLL_NAME", *Condition, *GenQInp);msiExecGenQuery(*GenQInp,*GenQOut);foreach(*GenQOut){ msiGetValByKey(*GenQOut,"COLL_NAME",*outcoll); writeLine("stdout", "*outcoll/hello"); }}INPUT *Condition="DATA_NAME = 'hello'"OUTPUT ruleExecOut

40

GenQuery Exampleirule –F test0.r

/compZone/home/leesa/hello/compZone/home/outgoing/hello/compZone/home/rods/hello/compZone/trash/home/rods/hints/linux-hints/ischia2-hints/hello

This rule was run as user rods (admin user) on compZone. GenQuery will not allow all users to make all queries. ACL permissions are respected with GenQuery.

Examples: see /compZone/home/rods/rules/genQuery or/snicZone/home/pracerods/rules/genQuery

Try tweaking and running some examples.

41

Database Resources• DataBase Resource (DBR): an external database (or similar

tabular information) that can be queried and updated via SQL statements (or other, for non-SQL)

• Database object (DBO): an interface to the DBR - typically a query that returns results

• DBOs typically contain SQL

• Query results can be stored to an iRODS data object, a DBO Results file (DBOR).

• idbo command – to run the DBO query

• access controls imposed by iRODS ACLs

• https://www.irods.org/index.php/Database_Resources and https://www.irods.org/index.php/Database_Resource_Administration

42

Setting Up a DBR• See

https://www.irods.org/index.php/Database_Resource_Administration

1. Install a non-IES server (an additional resource) on a machine that has access to the remote DB. Example: install on iren.renci.org.

2. On iren.renci.org, edit server/config/dbr.config - add a line containing: DBR-name DBMS-username DBMS-password DB-type (postgres, oracle,

mysql) Example: leesabase leesa leesapw postgresql

3. On iren.renci.org, put a .odbc.ini file into $HOME[IRODS_DBR_leesabase]

Driver=/home/leesa/tutorial-testing/pgsql/lib/libodbcpsql.so.2.0.0 Debug=0 CommLog=0 Servername=informatics.renci.org Database=leesabase ReadOnly=no Ksqo=0 Port=5432

(Can copy – mostly – from $HOME/.odbc.ini

on your data grid server.)

43

Setting Up a DBR4. Edit iRODS/config/config.mk:

– uncomment DBR=1– define POSTGRES_HOME:

POSTGRES_HOME=/home/leesa/tutorial-testing/pgsql

5. Make sure the LD_LIBRARY_PATH includes the postgres libraries: setenv LD_LIBRARY_PATH /home/leesa/tutorial-testing/pgsql/lib

6. Do a new "make" of the code in iRODS/and then ./irodsctl irestart

44

Setting Up a DBR7. Go over the main data grid (ischia2.renci.org) and use iadmin to

define the DBR in the ICAT. See 'iadmin h mkresc’. mkresc Name Type Class Host [Path] (make Resource)

Example: mkresc leesabase database postgresql iren.renci.org

/home/leesa/tutorial-testing/iRODS3.1/Vault

8. Give ownership of the DBR to the appropriate user(s) - for write/query permission to the DBR: 'ichmod -R own User-name DBR-name’Example: ichmod -R own rods leesabase

9. See iRODS/server/icat/dbr, containing example DBOs. Put a query into a file and ingest into iRODS: iput lt.pg

45

Setting Up a DBR10. Use isysmeta to declare this as a DBO type: isysmeta mod lt.pg datatype 'database object'

11. Use idbo to execute a query (contained in a dbo) on the DBR (leesabase in this case), running as a user that has ‘own’ permission on the DBR: idbo exec dbrname dboname

Example: idbo exec leesabase lt.pg

Note: when giving this command, you either need to be in the collection that contains the dbo file, or you need to give the full path name to it.

46

Using a DBR• idbo to execute a query contained in a DBO

• Store results into a DBO Result file (DBOR) by using idbo as an interactive shell:>idboidbo> output lt.out'exec' results will be stored in

/compZone/home/rods/dbr-examples/lt.out

idbo>exec leesabase lt.pgOutput written to /compZone/home/rods/dbr-examples/lt.outidbo>quit

• Examples: in /compZone/home/rods/dbr-examples NB: For this tutorial the DBR hasn’t been set up on snicZone, so

DBOs must be run on compZone.

47

III. Complex Rules and Scheduling

48

Delayed Execution• ExamplemyTestRule{ delay("<PLUSET>1m</PLUSET>"){

writeLine("stdout",”Writing message with a delay."); msiSendStdoutAsEmail(*Mailto, "Sending email"); }

}INPUT *Mailto="leesa@renci.org"OUTPUT ruleExecOut

• Queue management:– iqstat– iqdel– iqmod

49

Periodic ExecutionExamplemyTestRule {# Input parameters are:# Source collection path# Target collection path# Optional target resource# Optional synchronization mode: IRODS_TO_IRODS# Output parameter is:# Status of the operation# Output from running the example is:# Synchronized collection 1 with collection 2# delay("<PLUSET>5m</PLUSET>EF>1h</EF>"){ msiCollRsync(*srcColl,*destColl,*Resource,"IRODS_TO_IRODS",*Status); writeLine("stdout","Synchronized collection *srcColl with collection *destColl"); }}INPUT *srcColl="/compZone/home/leesa/tutorials",

*destColl="/compZone/home/leesa/tutorials2", *Resource="demoResc"OUTPUT ruleExecOut

50

More Interesting RulesPolicy to control where data is stored upon ingestion (courtesy of iPlant)

• Data ingested into TACC collections will be forced to resource “corral”: acSetRescSchemeForCreate { ON($objPath like "/tacc/Collections/*") {msiSetDefaultResc(corral,forced);} }(This forces physical location, with no effect whatsoever on logical location.)

• All other data goes into iplant resource group, randomly distributed among resources there:acSetRescSchemeForCreate {msiSetDefaultResc("iplantRG","preferred"); msiSetRescSortScheme("random"); }(This means that a default resource doesn’t need to be specified by users.)

51

Interesting RulesRule to load metadata on a file

(courtesy of Reagan Moore)

myRulemsiAddKeyVal {#Input parameters# Key-value buffer (may be empty)# Key# Value msiAddKeyVal(*Keyval,*Key,*Val); msiAssociateKeyValuePairsToObj(*Keyval,*FilePath,"-d"); msiGetDataObjPSmeta(*FilePath,*Buf); writeBytesBuf("stdout",*Buf);}INPUT *FilePath="/lifelibZone/home/rwmoore/foo1", *Key="School",

*Val="INLS"OUTPUT ruleExecOut

52

Interesting RulesSet default resources and enforce size limits on IDRISZone1(courtesy of Anges Ansari)

acSetRescSchemeForCreate {ON($objPath like "/IDRISZone1/*") { if (double($dataSize) <= 104857600) {msiSetDefaultResc("Resc1","null")} else {msiWriteRodsLog("File size not allowed","status");fail; } }

ON($objPath like "/ccin2p3/*") { msiSetDefaultResc("lyon2","null");

} }

53

Complex Rule – Verify integrity and number of replicas

schedulerReplicas {# This rule requires iRODS version 3.1 (msiCloseGenQuery mods)# The replicas for each file are updated to the most recent version# Each file is checked to verify existence of all required replicas and validity of checksums# As replicas are created, the algorithm round robins through available storage vaults# Checks that the number of storage resources used within a collection is greater than or# equal to the number of desired replicas.# This uses a just in time scheduler that slows down the processing rate# to complete the task within the specified number of seconds (*Delt)# Checks a TEST_DATA_ID parameter associated with the collection# to determine enable restarts after system interrupts# Writes a log file stored as Check-Timestamp in directory *Coll/log# Get current time, Timestamp is YYY-MM-DD.hh:mm:ss msiGetSystemTime(*TimeS,"unix"); msiGetSystemTime(*TimeH,"human"); *NumBadFiles = 0; *NumRepCreated = 0; *NumFiles = 0; *Runsize = double(0); *Sleeptime = 0; *colldataID = "0";

(Courtesy of Reagan Moore)

54

Verify integrity and number of replicas(continued)

# This is used to round robin through available storage resources *Jround = 0;# Check whether a collection was defined msiIsColl(*Coll,*Result, *Status); if(*Result == 0 || *Status < 0) { writeLine("stdout","Input path *Coll is not a collection"); fail;}#============ create a collection for log files if it does not exist

========== *LPath = "*Coll/log"; msiIsColl(*LPath,*Result, *Status); if(*Result == 0 || *Status < 0) { msiCollCreate(*LPath, "0", *Status); if(*Status < 0) { writeLine("stdout","Could not create log collection"); fail; } }

55

Verify integrity and number of replicas(continued)

# create file into which results will be written *Lfile = "*LPath/Check-*TimeH"; *Dfile = "destRescName=*Res++++forceFlag="; msiDataObjCreate(*Lfile, *Dfile, *L_FD);# check whether the attribute TEST_DATA_ID has been set from a priorexecution *Val = "0"; msiExecStrCondQuery("SELECT COUNT(META_COLL_ATTR_NAME) where COLL_NAME

='*Coll' and META_COLL_ATTR_NAME = 'TEST_DATA_ID'",*GenQOut2); foreach (*GenQOut2) { msiGetValByKey(*GenQOut2, "META_COLL_ATTR_NAME", *Val); } if(int(*Val) == 0) { *Str1 = "TEST_DATA_ID=0"; msiString2KeyValPair(*Str1,*kvp); msiAssociateKeyValuePairsToObj(*kvp,*Coll,"-C"); writeLine("*Lfile","added TEST_DATA_ID attribute to collection *Coll"); }

56

Verify integrity and number of replicas(continued)

# on a restart TEST_DATA_ID will be greater than 0 msiMakeGenQuery("META_COLL_ATTR_VALUE", "COLL_NAME = '*Coll' andMETA_COLL_ATTR_NAME = 'TEST_DATA_ID'",*GenQInp2); msiExecGenQuery(*GenQInp2,*GenQOut2); foreach(*GenQOut2) { msiGetValByKey(*GenQOut2, "META_COLL_ATTR_VALUE",*colldataID); }# *colldataID is the string identifier of the last file that has beenchecked msiCloseGenQuery(*GenQInp2, *GenQOut2); msiMakeGenQuery("count(DATA_NAME), sum(DATA_SIZE)","COLL_NAME = '*Coll'and DATA_ID > '*colldataID'", *GenQInp2);# this counts all files that have not yet been checked including replicas msiExecGenQuery(*GenQInp2, *GenQOut2); foreach(*GenQOut2) { msiGetValByKey(*GenQOut2, "DATA_NAME", *num); msiGetValByKey(*GenQOut2, "DATA_SIZE", *sizetotal); }

and on for manymore pages…