File: 05_RETS_Software_Development.PPT 1 RETS Software Development RETS the standard RETS...

File: 05_RETS_Software_Development.PPT 1

RETS Software Development


RETS the standard

RETS transactions– Identify issues, concerns and recommendations

Sample session workflow



RETS Software Development Topic Objectives

Improve understanding of the RETS architecture

Build familiarity with the RETS document

Enable more productive RETS software development



RETSPrinciples - 1

Stateless communications protocol

Carry state using a cookie:– Basically a token that points to a persistent store on a

server somewhere

– RETS defines an optional cookie for this purpose

Recommendations:– Return all cookies received

– Client applications using browserswill need to accept cookies



RETSPrinciples - 2

Uses Http-based transport

Uses DMQL - an open query language

Supports XML and site-specific output data formats

Supports site-specific metadata to describe the local MLS system

Recommendation:– Effort to support metadata produces interoperability

benefits



RETS Document

Organization follows a typical real estate industry workflow:– Initiate single session

– Multiple data requests

– Terminate session

References standards produced by other standards organizations

RETS document

can be challenging to read



Simple Collaboration Diagram

<<actor>>Client

LoginHandler

GetMetadataHandler

MetadataCache

SearchHandler

<<actor>>RetsServer

SearchResults

SearchDisplay

SearchSelectorDisplay

ActionUrlDisplay

LoginDisplay

1.1: SendCredentials

4.1: PrepareSearch

4.1.1.1: Results

3.1: Metatdata Results

2: CapabilityUrls

4.1.1: DisplayResults

5: Search

3: GetMetadata

2.2: GetMetadata

2.1:

4:

1: Login



Login Transaction Making the Initial Connection

Authentication– Response with authorized resources

Types:– Basic

– Digest using MD5 or SHA1

– SSL

Recommendations:– Implement Digest using MD5

– SHA1 if you can



Login TransactionAuthentication

Most common authentication type is digest using MD5– Somewhat tricky to implement algorithm

– Easiest to borrow the implementation from one of the open source projects

When in doubt:– Ask the RETS-DEV list

– Ask your software vendor



Login Transaction Activity DiagramRETS ServerRETS Client

ReceiveLoginRequest

ValidateMD5

BuildSession FetchCapabilityURL

ResponseLogin

InternalActivity

RequestLogin

Construct MD5

ValidSession

Valid header

Invalid header

<

>



Login TransactionDigest Authentication

Message Digest algorithms are one-way functions

Easy to perform the calculation

Difficult to reverse the calculation

Both sides calculate the same message digest

Transmit only the digest

Compare your calculated digest with the received digest



Login TransactionUserAgent

Application name accessing the MLS server– All client requests MUST include this field

– This is a standard HTTP header field as defined in RFC 2616

MLS server should validate application name presented against list of allowable applications maintained on a table

Application name and userid is a useful key to a table that tracks MLS server usage



Login TransactionGotchas

Using operating system libraries for MD5 can cause authentication failure

Proxies and tunnels can be a problem

Re-authenticate is part of the RFC but few vendors enforce it

Case may matter



Sample LoginChallenge Response Header 1

HTTP/1.1 401 Unauthorized xxx

Date: Mon, 02 Aug 2004 05:23:54 GMT

Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with

WWW-Authenticate: Digest realm="[email protected]", nonce="313035", opaque="6e"



Sample LoginChallenge Response Header 1 cont.

Content-Length: 0

Content-Type: text/html

RETS-Version: RETS/1.0

Cache-Control: private



Sample LoginNonce

nonce=

"313035393731353435373630342058dd631265e3360724c45d15f23aec7a"



Sample LoginOpaque

opaque="6e6f742075736564"



Sample LoginChallenge Response Header 2

HTTP/1.1 200 OK

Date: Mon, 02 Aug 2004 05:31:03 GMT


Content-Type: text/plain



Sample LoginChallenge Response Header 2 cont.

Set-Cookie: RETS-Session-ID=20000439936


Transfer-Encoding: Chunked


0230



Sample LoginChallenge Response Body 2

<RETS ReplyCode="0" ReplyText="Success">

MemberName=Mark Crawford

User=632344, NULL, NULL, 632344

Broker=National,1

MetadataVersion=1.2.0

MinMetadataVersion=1.1.7

OfficeList=National;1

TimeoutSeconds=900



Sample LoginChallenge Response Body 2 cont.

Search=http://rets.ftc2.com:6103/search

GetObject=http://rets.ftc2.com:6103/get object

Login=http://rets.ftc2.com:6103/login

GetMetadata=http://rets.ftc2.com:6103/getmetadata

Logout=http://rets.ftc2.com:6103/logout

</RETS>

0000



Login Transaction Capability URL List

Response from the Login transaction includes a list of known resources on the server

– Each URL points to a RETS transaction type

Capability URL list may be different based on the user and their authorization level

Required that the server return a list containing at least Search

The URL references are not permanent locations– the standard allows the location to change



Action-URL and Get Transaction

Included in the Capability URL List is an Action-URL

Points to a human readable message – Most useful if changed regularly; perhaps daily

Immediately following successful login, may need to do a Get on the Action-URL– Get Transaction is a file retrieval from the server



Metadata

Description of the structure of the data on a specific server

Unique to each site– There will be common elements

Most powerful feature of the RETS specification

Perhaps the most challenging part

of the RETS specification



GetMetadata Transaction Overview

Transaction to retrieve system metadata

Can select a specific level, the level and descendants or a specific metadata ID

Formats supported are COMPACT and Standard-XML

Metadata id value of 0– Request is for all types at the same level – breadth

Metadata id value of *– Request is for all types at the same level and all child types -

depth



Metadata FormatID

Hierarchy of types of metadata– See Figure 11.1 on page 11-2

of the RETS specification

When referencing hierarchy as ID– Parent: Child is the form

– Example:

Type: METADATA-UPDATE_TYPE

ID: Property: RES: Add



Metadata FormatVersioning - 1

Hierarchy of metadata is versioned

Changes to lower level of the hierarchy propagate upward– Example: Changed METADATA-TABLE attribute will

propagate up through CLASS, RESOURCE and SYSTEM



Metadata FormatVersioning - 2

Caching of metadata is permitted

Use the login information to determine metadata changes

Recommendation:– If possible, get all the metadata on a change instead

of attempting to get a specific type



Metadata FormatVersion 1.5 changes

Added the Foreign Keys metadata– Provides relationships between the offered resources

of the form parent:child

Clarified a number of issues with Metadata Format

Recommendation:– Use 1.5 as a reference even when implementing

1.0.1 compliant software



Application Responses to Metadata Changes

Ignore metadata– Change breaks the application, intervention required

to fix

Static metadata– Change is invisible, applications keep working,

intervention to required view change

Dynamic metadata– Change is visible to applications without intervention



Search Transaction Overview

Most important RETS transaction

Can use either http GET method or http POST method– Example – GET method request

http://rets.server.com:6103/search?Class=RES

Issue: GET method is limited in the character length of the request

Recommendation: POST method



Search TransactionClient Diagram

MetadataCache

SearchDisplay

SearchSelectorDisplay

<<actor>>Client

<<actor>>RetsServer

2: Query response

1: query selection

1.1: Http query request

2.1: result display



Search TransactionServer Diagram

<<controller>>SearchHandler

<<view>>SearchResults

<<model>>RepositoryModel

<<actor>>Client 1.1.1:

1:

2:

1.1:

Server side view.Implemented as MVC



Sample SearchSearch Request Header

HTTP/1.1 200 OK

Date: Fri, 31 Jul 2004 23:22:43 GMT


Content-Type: text/xml



Sample SearchSearch Request Header cont.

Set-Cookie: RETS-Session-ID=20000439934

Set-Cookie: RETS-Request-ID="42"


Transfer-Encoding: Chunked




Sample SearchSearch Response Body

0fe8

<?xml version="1.0" ?>

<!DOCTYPE RETS SYSTEM "http://www.ftc2.com/rets/dtd/RETS-20010812.dtd">

<RETS ReplyCode="0" ReplyText="Success">

<COUNT Records="2507" />



Sample SearchSearch Response Body cont.

<REData>

<Properties>

<ResidentialProperty>

<Building>

<BuildingDescription>

<Stories>1</Stories>

<PropertyCondition>Shows Well</PropertyCondition>




Search TransactionSearchType Required

SearchType: The resource id to search on– Example – SearchType=Property



Search TransactionClass Required

The class of data within the resource to search against– Example – Class=RES

Issues:– Some of the data may be in system specific Class

names rather than the Class name that represents an XML Standard Name



Search TransactionQueryType Required

QueryType: The query language– Only two types: DMQL and DMQL2

– Example – QueryType=DMQL2

Issues:– Mainly stylistic difference, DMQL2 supports

quoted text

– Not suitable for highly complex searches

– Not possible to perform cross Class searchesExample – NOT Residential and Lot Land



Search TransactionQuery Required

Query:– Key – value pairs

– Uses field name based on metadata or on standard names

– Example

Query=(ListPrice=200000-300000),(Status=|A)

Issues:– Lookup type queries can execute for a long time



Search TransactionOptional Arguments

Optional argument are just that

A vendor MAY choose to implement none, one or more

Software is RETS compliant without any of the optional arguments implemented

Issue:– Mismatch between client & server feature sets

Recommendation:– Do NOT assume, verify the feature set



Search TransactionCount

Default is no count

Behavior outside of given three values not defined



Search TransactionFormat

Return records in XML or column header/row values format– XML is more verbose, but contains structure

information

– Some issues with site specific XML – clarification in RETS needed

Default is Standard-XML

Issue:– Site specific XML is not defined in the current

standard



Search TransactionFormat Sidebar: XML - 1

XML is a meta-language that defines a markup language

RETS is an example of an XML application

An application consists of a DTD and documents that obey the rules embodied in the DTD

The XML DTD is a document that describes the structure and elements of an XML document



Search TransactionFormat Sidebar: XML - 2

The XML document takes data and applies the DTD

RETS will eventually move from a DTD based vocabulary to a Schema-based vocabulary



Search TransactionFormat Sidebar: Compact Formats

Provides a delimited column description

Provides delimited data in the same position as the column description

Field delimiter is chosen by the server

Gotcha:– Delimiter tag is a two character representation of the

ASCII character

– Delimiter value may collide with data values in earlier implementations



Search TransactionLimit

Limits the number of records returned:– The NONE well known name requests a suspension

of limit

– Behavior is implementation dependent

– Requesting a Limit should make the query faster

– May still search for all records but only transmit records to a Limit



Search TransactionOffset

Return records starting from the specified point in the result set:– One method to handle very large data pulls

– Given that the underlying transport mechanism is not guaranteed, very large results can fail with transport errors

– Given that the data volumes can be very large, out of memory errors can occur



Search Transaction StandardNames

Indicates that the query uses the standard names or the system names:– Servers are not required to implement standard

names to meet RETS compliance criteria

– Standard Names are intended to be common across all RETS systems

Refining the list of Standard Names

is an ongoing process



Search TransactionRestricted Indicator

Certain fields may have restricted visibility:– Client may request fields that are restricted be

replaced by a specific character string

– Example - compensation amount field

– Client may request that this amount be masked by ###

– Server default is to return a null value



Search TransactionSelect

Provides the ability to create a subset of the fields accessible to the client

Default is to return all accessible fields

Gotcha: Select field names are not specified as either the system names or the standard names– Ambiguity in RETS standard

– Working assumption is that the server will understand both



Search TransactionDMQL

Defines the search criteria for the search result set

Based on a BNF grammar to provide a neutral representation

RETS Server parses the DMQL and creates a server specific query

Trade off between domain specific query forms and more complex query grammars like SQL-99

Issues around the lexical parsing of the grammar



Update TransactionOverview

Allows addition of records to the RETS system

Supports Auto-population of fields

References only by System Name, not Standard Name

Validation based on rules described in the metadata

Single record per update – no bulk update



Update TransactionValidation

Basic validation on data types

Additional validation for single lookup, multi-lookup and range

Additional validation for Boolean expression

Additional validation using external validation

Certain validation steps may be server only



Update TransactionError Block

Validation tests that fail result in an error block that provides some information to assist in correction

One error block per failed validation test– Means a single field may trigger multiple errors



Update TransactionGotchas and Recommendations

Gotcha:– RETS 1.7d1 adds a third state to the validate-flag “2”

meaning “validate the record but do not save it”

– Be aware of this coming specification change

Recommendation:– Client-side validation is fast fail – use it



GetObject Transaction Overview

Provides a way for users to get objects defined in the metadata

Main use is to return pictures although there are other media types

Returns the object in the appropriate mime type format - this may be a binary format

Recommendation:– Ordering of the object ID changed in 1.5D2– Implement using the most recent RETS version



ChangePassword Transaction Overview

Allows a user to change their password

Some ambiguity in the 1.5 and earlier standard

Cryptographically weak

Recommendations:– Be very careful when implementing

– Use the 1.7 standard addition as described in RCP 041



Logout Transaction Overview

Courtesy transaction

Communicates that a user has finished the session

A mechanism for the server to:– dispose of any local resources

– communicate session and billing information

Recommendation:– Clients should log out

– Servers should be aware that clients may never log out



Extensions to the RETS Standard

RETS allows vendors and sites to extend the standard to meet the specific needs of a site

Additional Capability List URLs have no suggested form

Additional parameters to an existing transaction and additional metadata have should have prefix ‘X’– Example: Add sort direction to Search: XSortDir=Asc

Almost everything else is covered by metadata



Role of Parser Generators

Used to produce parsers for:– DMQL– Validation Expressions

DMQL Java code will be used by RETS server to generate SQL from DMQL

Validation expression Java code will be used by RETS server to execute the validation expression



Benefits for using a Parser Generator

Improves productivity for software developers– Not easy to write a good parser

– Relatively easy to write the grammar input file for the parser generator

Avoids having to develop a parser– Difficult, demanding, time-consuming

Makes use of available parser generators– Examples: javacc, ANTLR, lexx

Produces more robust Java source code



Generating Java Source Codefrom a BNF Grammar File

BNF Grammar File(produced by

developer)

JavaccParser Generator

JavaSource code

BNF grammar for DMQL - specified in RETS specification- see example on next two slides

Grammar file syntax - specified in parser generator specification- see example on next two slides



BNF Grammar File Fragment DMQL Javacc Input File - 1

PARSER_BEGIN(DMQL)

// The DMQL Javacc file which is used to generate the the primary class for parsing DMQL queries

public class DMQL

{<Your Java code here>}

PARSER_END(DMQL)

// Parse the DMQL statement.

void Input() :

{< Java attributes>}

{<Javacc expressions & your Java code>}

// Match the the DMQL Query at the highest level.

void Query() :

{< Java attributes>}



BNF Grammar File Fragment DMQL Javacc Input File - 2

{ Search() (<EOF> | <TERMINATOR>)}

... BNF and Javacode Productions and additional Java code for processing tokens for parsing DMQL based on the RETS DMQL Grammar...

// Terminals (Tokens)

TOKEN : /* The DMQL Tokens (See RETS Specification) */

{

< PLUS: "+" >

| < SINGLE_CHAR: "?" >

| < MULTI_CHAR: "*" >

.

| < OR: "|" >

}



Application ofParser Output

Java code toprocess DMQL

SQL

Input rules for update(produced by

MLS)

Java ValidationExpression Codeto evaluate Rules

ValidationOutcome

Input DMQL(produced by

client software)

ParserGenerator

Output



Resources forParser Generators

ANTLR, ANother Tool for Language Recognition– http://www.antlr.org

Java Compiler Compilertm – JavaCCtm

– https://javacc.dev.java.net

Lexers and Parser Generators – http://catalog.compilertools.net/lexparse.html



TopicSummary

RETS provides a rich set of features for implementing a service or client

Careful design will result in a successful project

Work with the vendor or customer to ensure that the correct features are built

Read the RETS document and watch for forward and reverse references

Use RETS 1.5 as asupplement to RETS 1.0.1– it includes many clarifications



SoftwareQuality



Top 10 signs TechnologyHas Taken Over Your Life

10. You need to fill out a form that must be typewritten, but you can't because there isn't one typewriter in your house - only computers with laser printers.

9. You would rather get more dots per inch than miles per gallon.

8. You rotate your screen savers more frequently than your automobile tires.

7. You think of the gadgets in your office as "friends," but you forget to send your father a birthday card.

6. You know Bill Gates' e-mail address, but you have to look up your own social security number.

5. On vacation, you are reading a computer manual and turning the pages faster than everyone else who is reading John Grisham novels.

4. The thought that a CD could refer to finance or music never enters your mind.

3. Your letterhead lists a phone number, a fax number, e-mail addresses for two online services, and your web site address but no street address.

2. You have never sat through an entire movie without having at least one device on your body beep or buzz.

1. You sign Christmas cards by putting :-) next to your signature.



TypicalSession WorkflowLogin

Search

Get Object

Logout

UpdateMetadata

?

Get Metadata

GetObjects

?

Data aboutavailable

serverdata

Data aboutOffices,Agents,Listings

Pictures,Video

Documents

File: 05_RETS_Software_Development.PPT 1 RETS Software Development RETS the standard RETS...

Documents

Transcript of File: 05_RETS_Software_Development.PPT 1 RETS Software Development RETS the standard RETS...