Intro to SAM DbServer Rewrite Toolkit and Components

Lauri Loebel Carpenter

November 2003

Last year, if you recall…

DbCorbaClient / SamApi

(or, what I did on my Thanksgiving vacation)

Lauri Loebel Carpenter

December 10, 2002

Where did this lead?

• Complete IDL redesign• Infrastructure to support new concepts:

– Struct Wrappers– Automarshalling DbServerProxy– Automarshalling DbServantImpl– DbAware Objects– ProtectedInterfaces– Exception handlers, db utilities, etc.

IDL: Interface Definition Layer

• Defines anything and everything that needs to be known by two or more code packages.– Constants

– Data Structures

– Available Interfaces

• Allows communication between arbitrary programming languages on arbitrary platforms– Powerful, but error-prone and difficult to debug.

Marshalling Data is complicated; Unmarshalling is

worse!• Marshal: take my object, convert it to a byte

stream and send it to another application– On a different platform

– In a different programming language

• Unmarshal: receive this arbitrary bag of bites, and figure out what it is and what it is supposed to do, and send back some results.

• And handle all of the exceptions.

Bottom Line:

We really do not want arbitrary pieces of code to be marshalling and unmarshalling data structures.

Encapsulate! StructWrappers

• For each IDL data structure, create a representation that knows how to marshal/unmarshal itself.

• Perform all marshalling/unmarshalling in one place (dbProxy, implCreation)

• Treat exceptions as just another type of object to be marshalled/unmarshalled

Struct Wrappers in a nutshell

• initialize_fromPython(self, *args, **kwargs)• initialize_fromCorba(self, corbaObject)• pyRepr()• corbaRepr()• __repr__()• __str__()

Struct Wrappers: the tricky part: Many initialization signatures.

• Valid SamSizes: s1 = SamSize('10g') s2 = SamSize(s1) s3 = SamSize(0) s4 = SamSize('100', 'bytes')

• Valid RunDescriptors: r1 = RunDescriptor(runId=10)

r2 = RunDescriptor(r1)r3 = RunDescriptor(runNumber=12, runType='monte carlo')r4 = RunDescriptor(12, 'physics')r5 = RunDescriptor('physics', 12)

Automarshalling Proxies/Servers:

# get a proxy to the interface of interest:dbProxy = proxyMgr.getProxy('DataFile')

# create the wrapped object representing the# high level data structuretheData = SamStructWrapper('input', 'args')

# pass the high-level data structure to the method; # result will be a high-level data structure alsoresult = dbProxy().interfaceMethod(theData)

Db (and other) Proxies:

• Standard SamServerProxyManager utility for managing proxies

• Marshals all args (via corbaRepr())• Unmarshalls all return values (via

pyRepr())

• Client code needs ONLY high-level (auto-documentable) data structures

ServantFactory and MethodWrappers

• ServantFactory creates an instance of an Implementation

• Then takes that Implementation and wraps the LOCAL, PUBLIC methods in a ServantMethodWrapper

• ServantMethodWrapper unmarshals all incoming data (via pyRepr()), marshals all return values (via corbaRepr())

• ServantMethodWrapper also handles marshalling/unmarshalling of exceptions raised during execution of the method

Implementation Layer:

• Receives the high-level data representation• No CORBA marshalling/unmarshalling code to

distract the developer from SQL nightmares.• Returns another high-level object• ServantMethodWrapper marshals it• dbProxy unmarshals it…

• Server code needs ONLY high-level (auto-documentable) data structures

Conceptual Example:What the User Sees

from SamFile.SamDataFile import SamDataFilerawFile = SamDataFile(fileName='demo.raw', fileType='colliderImport', startTime='10-oct-2003', endTime='12-oct-2003', runType='physics data taking', runNumber=2001, firstEvent=1, lastEvent=10, eventCount=10, datastream='physical generic', dataTier='raw', filePartition=200, appName='datalogger', appFamily='datalogger', appVersion='0.0')fileId = rawFile.declareMetadata()

What the client code looks like (simplistically):

class SamDataFile(DataFileAttributes):

….

def declareMetadata(self):

proxy = self.proxyMgr.getProxy('DataFile')

fileId = proxy().declareMetadata(self)

self.setFileId(fileId)

What the Server code looks like(very simplistically):

class DbAwareSamDataFile(SamDataFile):

….

def declareMetadata(self):

self.validateMetadata()

fileId = self.dbCore.nextId('data_files_id_seq')

self.dataFiles.insertOne({'fileId':fileId,

'fileName' : self.getFileName(),

'fileSizeInBytes': self.getFileSize().getSizeInBytes(),

…. } )

Implementation: Summary

• Define data structure– Sam_idl/sam_corba_base/– Sam_common_pkg/SamStruct/

• Add methods– Client: calls dbproxy().method– Server: DbAware object implements the guts

of the method

Other DbServer Tools and Utilities:

• Protected Interface/Implementation– Samadmin, shifter commands

• nameAndIdUtility– Standard utilities for NameOrId objects

• relationshipUtil– Standard utilities for validating relationships

• DbServer ExceptionHandler– Raise ONLY what is declared in the IDL; notify

developer of any undeclared exceptions and convert them to InternalError

Other General Tools and Utilities:

• SamServerProxyManager– Maintains all proxies

• SamExceptionHandler– Exception auditor– formattedTracebackString– Additional information in argDict

Where to find more info:IDL

Sam_idl/

/sam_corba_base/Typedef.idl

/sam_corba_base/<dataStruct>.idl

/sam_db_srv/<interface>.idl

Where to find more info:Struct Wrappers

Sam_common_pkg/

SamCorba/SamIdlStructWrapperBase.py

SamCorba/SAMToSamStructCodeGenerator.py

Sam_common_pkg/

SamStruct/<dataStruct>.py

Where to find more info:Automarshalling

Sam_common_pkg/

SamCorba/DbServerProxy.py

Sam_db_srv/

src/ServantMethodWrapper.py

Sam_common_pkg/

SamUtility/MethodWrapper.py

Where to find more info:Misc.

Sam_common_pkg/

SamException/SamExceptionHandler.py

SamExceptions.py

Sam_common_pkg/

SamFile/SamDataFile.py