Technical Track Session Data Transport Standard Nathan Chitty - PESC Gary Sandler - PESC.
-
Upload
lisa-reynolds -
Category
Documents
-
view
215 -
download
0
Transcript of Technical Track Session Data Transport Standard Nathan Chitty - PESC Gary Sandler - PESC.
Technical Track Session
Data Transport Standard
Nathan Chitty - PESC
Gary Sandler - PESC
2
Data Transport Standard - DTS
• DTS uses Internet technologies to facilitate real time data exchange and transaction processing
• DTS builds on stable technologies, not specific products
• DTS, once implemented, reduces programming and per-transaction costs through standardization
3
DTS Defined
• Data Transport Standard is a specification not a product
• Established by Postsecondary Education Standards Council (PESC) for exchanging data for:– Inquiries– Reports– Transactions
4
DTS Defined
• A specification for an adjunct to or a replacement for existing data transport mechanisms– PGP / GnuPG encryption
– SecretAgent w/ Email
– FTP and SecureFTP
5
DTS Benefits
• A Web Services implementation– Delivery confirmation included – no
guessing– All requests get a response– All submissions get an answer of
some kind
• Facilitates real time data exchange
6
DTS Benefits
• Includes automatic data encryption
• Uses digital signature standards
• Platform independent
• Strong authentication with non-repudiation
7
Benefits To “System Providers”
• Add value to schools’ systems– Schools want transport added to systems
and are generally not concerned with the technologies
• Easier to build one transport protocol for all recipients – Just as CommonRecord created the drive
to build one XML format
8
Benefits to “Service Providers”
• As everyone implements DTS, the need to support other transports will drop
• If any school implements DTS, service providers will have to support it
• Also provides a single communication infrastructure option for internal systems
9
DTS Specification
• Specification covers– Technical interchange rules and processes– Recommended best practices
• Technical Specification is the pure Simple Object Access Protocol (SOAP) interface
• Implementation Guide is for both .Net and Java reference implementations
10
DTS Specification
• Reference implementation examples are available
• Specification does not cover– Business rules for transaction
processing– Operational oversight, monitoring or
escalation
11
Data Transport Issues in Higher Ed• E-mail is not reliable or flexible
enough– No guarantee of delivery– No guarantee of order of delivery for
sequence dependent data– No automatic confirmation of receipt
or facility for retransmit
12
Data Transport Issues in Higher Ed• E-mail is not reliable or flexible
enough (continued)
– No synchronous response available
– Email size limitations
13
Data Transport Issues in Higher Ed• FTP data exchange has own
challenges– Possible to overwrite earlier files
– No confirmation of receipt
– No synchronous response
14
Data Transport Issues in Higher Ed• Encryption today is always
separate and subject to its own – Issues
– Maintenance
– Failures
15
DTS Addresses These Transport Issues• DTS addresses
– The confirmation issue with a send-receive protocol – confirmation is built in
– The order of delivery problem by actively delivering and receiving the data – no unconfirmed hand-offs
16
DTS Addresses These Transport Issues• DTS addresses
– The size problem through data compression
– The FTP overwrite problem by not using filenames
17
DTS Addresses These Transport Issues• DTS addresses
– The lack of a synchronous response by building in a required synchronous response, even if only for handling status
– The encryption issue by using standard HTTPS for encryption – the same technology as for online banking
18
DTS Technical Workgroup
• Task: Create a written specification for real-time exchange of data between organizations– Meets business requirements– Standards based– Standard technologies (Java, .Net)– Payload Insensitive– Secure and reliable
19
DTS Technologies
• Global XML Web Service Architecture (GXA), generally accepted as the foundation for building Web Services– WSDL (Web Service Definition
Language)– SOAP (Simple Object Access Protocol)– WS-I (Web Service Interoperability)– WS-S (Web Service Security)
20
DTS Technologies
• WS-Security (Digital Signatures)– Strong authentication with non-
repudiation
– X.509 encryption keys and certificate authorities
• SSL encryption of HTTP streams
21
Anticipated Architectures
• Immediate processing– Request and processed Result Response
• “Push/Push” deferred processing– Request and Acknowledge Response– Request with Result and Acknowledge Response
• “Push/Pull” deferred processing– Request and Acknowledge Response (just send)– Request for Result and Result Response
22
`
SBS/FAMSDTS Client Enabled
DTS ServicesubmitDTS() Back-end Processing
Central Repository (CR)
Key/URL/Services
Data Transport Standard: Immediate Processing DTS implementation
2. submitDTS invoked against enpoint
4. Response: Actual processed response to payload<DTSAcknowledge>Immediate</DTSAcknowledge>
This configuration could have a server acting as a DTS Client that received
data for transport from the back end.
Data passed to back-end for processing
Processed result returned to be sent as response
Data passed to back end for processing.
Immediate
23
`
SBS/FAMSDTS Client Enabled
DTS ServicesubmitDTS()
Back-end Processing
DTS ServicesubmitDTS()
Server asDTS Client
Central Repository (CR)
Key/URL/Services
Data Transport Standard: Deferred Processing DTS implementation where originator is running a DTS Service to accept the processed result.
1. Client contacts CR to obtain Endpoint for
recipient per service
2. submitDTS invoked against enpoint
4. Acknowledgement- <DTSAcknowledge>Received</DTSAcknowledge>
3. Service contacts CR to obtain public key
of sender to authenticate and authorize
Data passed to back end for processing.
Response Data passed to transport for delivery
Validation could be performed by the back-end system in a synchronous
manner allowing errors to be reported back immediately
Payload of acknowledgement may or may not contain a “token” as the payload. In this
scenario it doesn’t make much sense (see deferred implementation without DTSService
at client site) for one to be returned
Central Repository (CR)
Key/URL/Services
3. Service contacts CR to obtain public key of
sender to authenticate and authorize
2. submitDTS invoked against enpoint
4. Acknowledgement- <DTSAcknowledge>Received</DTSAcknowledge>
1. Client c
ontacts CR to
obtain endpoint for
sending back
the Result o
f proce
ssing to
recipient
Exact same repository as the CR at top of page
Data passed to back end for processing.
This configuration could have a server acting as a DTS Client that received
data for transport from the back end.
Processing performed in the middle of accepting request and delivering processed result- Time frame of this step is immaterial
“Push/Push”
24
`
SBS/FAMSDTS Requestor
DTS RespondersubmitDTS()
Back-end Processing
DTS RespondersubmitDTS()
Central Repository (CR)
Key/URL/Services
Data Transport Standard: Deferred Processing DTS implementation where originator is ONLY running a DTS Client (a web service is not implemented for receiving the result)
1. Client contacts CR to obtain Endpoint for
recipient per service
2. submitDTS invoked against enpoint
4. Acknowledgement- <DTSAcknowledge>Received</DTSAcknowledge>
3. Service contacts CR to obtain public key
of sender to authenticate and authorize
Data passed to back end for processing.
Response Data passed to transport for delivery
Validation could be performed by the back-end system in a synchronous manner allowing errors to be reported back immediately
Payload of acknowledgement may or may not contain a “token” in the payload. If the token is present it is expected back as-is. The client is not expected to process the “token” in any manner thus allowing the “token” to be whatever the provider deems necessary.
Central Repository (CR)
Key/URL/Services
2. submitDTS invoked against enpoint; <DTSPayloadType>DTSRetrieve</DTSPayloadType>
4. Response- processed result returned as payload
3. Servi
ce co
ntacts CR to
obtain public ke
y of
sender to
authenticate and authoriz
e
Exact same repository as the CR at top of page
Data passed to back end for processing.
This configuration could have a server acting as a DTS Client that received data for transport from the back end.
Processing performed in the middle of accepting request and delivering processed result- Time frame of this step is immaterial
`
DTS Requestor
If token was received in response to original request and client wants that specific response, the token must be passed as payload. Otherwise, first response available for that client would be returned. Subsequent “DTSRetireve” would get next available response.
1. Client contacts CR to obtain endpoint to retrieve
processed results.
Push/Pull”
25
DTS Analogy
• DTS is the definition of the “Pipe” and the structure of its contents– The “Pipe” is the internet
– The content is SOAP
– The end points/junctions are Web Services
– The sources are Web Service enabled clients
26
DTS Analogy
• DTS defines how others can connect to the “Pipe” already installed– Any connections must have certain
“threads”
– Any connections must handle two way traffic independent of how the traffic will be used
27
DTS Analogy
• By knowing about the pipe and the type of connections, any “plumber” can use his/her own tools to make connections; just so long as the threads match
28
Extending the Analogy
• We all have plumbing and fixtures• Very unlikely we all have the same
type of fixtures• Yet our water companies still deliver to
us all• All our fixtures use (“process”) it• All our drains return it
29
How Did We Do It?
• Created basic HelloWorld service and client– Worked interoperable
• Added simple Headers to HelloWorld– Was not interoperable
• Added complex Header to HelloWorld– Was not interoperable
30
Why SOAP Headers
• To answer routing and processing expectations without opening the payload
• Remain payload insensitive
• Allow extensibility for new processes
31
DTS SOAP Headers• DTSRequestRouting• DTSRequestServiceExpectation • DTSRequestPayloadType• DTSRequestSignature• DTSResponseRouting• DTSResponseAcknowledge• DTSResponsePayloadType• DTSResponseSignature
32
Convoluted Filename vs Header Elements• A [B] <X.Y.Z:M>
A = File Type, B = Encrytption, X.Y.Z = key identifier, M = Unique message ID
• Encryption unnecessary because using HTTPS• DTSRequestPayloadType = A• DTSRequestRouting
– SourceIDSubCode = X, SourceID =Y(.Z)– UUID = M
33
Interop Problem with SOAP Headers• xsi:type attribute in Header
elements– Java includes and requires this
attribute
– .Net does not
34
All about SOAP
<soap:Header><DTSRequestPayloadType xsi:type="DTSRequestPayloadType" xmlns="http://www.datatransportstandard.com">
<value>CRC01Request</value>
</DTSRequestPayloadCode>
35
SOAP is the Key
• The SOAP transmitted across the wire is of primary importance– Element names– Type attribute– Not Namespace moniker (Java uses one
by default, .Net does not)
• How you get the correct SOAP is not important
36
Java:<ns1:DTSRequestSignature
soapenv:mustUnderstand="0" xsi:type="ns1:DTSRequestSignature"
xmlns:ns1="http://www.datatransportstandard.com"><ns1:value>SignatureValue</ns1:value>
</ns1:DTSRequestSignature>
.Net:<DTSRequestSignature
xsi:type="DTSRequestSignature" xmlns="http://www.datatransportstandard.com">
<value>SignatureValue</value></DTSRequestSignature>
SOAP Differences That Do Not Matter
37
Reference Implementation Architecture
• Client Application
• Client Core
• Service Core
• Service Application
38
Client Application
• Knows nothing of SOAP or Web Services
• Implements Client Core Interface– “Setters” and “Getters” of DTS
specific elements
• Houses specific business logic
39
Client Core
• Knows nothing of business logic• Uses properties set to construct the
SOAP• Interface for “setting send” and
“getting returned” elements• Handles the communication to Service
Core- DTS Specification
40
Service Core
• Accepts transmissions from Client Core• Implements Service Application Interface
– “Setters” and “Getters” of DTS specific elements
• Creates return SOAP – Format return acknowledgement or data from
Service Application– Construct SOAP faults
41
Service Core (continued)
• Isolated business logic– Examples
• Invoke Service Application based on payload
• Place payload in “queue”
42
Service Application
• Interface for “setting sent” and “getting to be returned” elements
• Houses specific business logic
• Knows nothing of SOAP or Web Services
43
Connecting the layers
Internet
Client Application
Client Core Service Core
Service Application
DTS SpecDTS Spec
Entity A Entity BPlatform specific communication
Platform specific communication
44
Connecting the layers
Internet
Client Application
Client Core Service Core
Service Application
DTS SpecDTS Spec
Entity A Entity BPlatform specific communication
Platform specific communication
Client Application
45
Connecting the layers
Internet
Client Application
Client Core Service Core
Service Application
DTS SpecDTS Spec
Entity A Entity BPlatform specific communication
Platform specific communication
Client Application
Client Application
46
Connecting the layers
Internet
Client Application
Client Core Service Core
Service Application
DTS SpecDTS Spec
Entity A Entity BPlatform specific communication
Client Application
Client Application
Service Application
Platform specific communication
47
Connecting the layersESB
Internet
Client Application
Client Core Service Core
Service Application
DTS SpecDTS Spec
Point A Point B
Client Application
Client Application
Service Application
Platform, but DTS interface Platform, but DTS interface
48
Additional DTS Information
• Visit PESC at www.pesc.org
• Materials available include– Executive summaries
– Specifications
– Reference (proof of concept) implementations
49
Adding SOAP Headers
• Change WSDL – Still has problems
• Create Container Classes
• Container Classes require serialization/de-serialization directives
50
Adding SOAP Headers
• Augment Service Code
• Augment Client Code
51
package com.datatransportstandard.www.serializable;
import java.io.Serializable;
public class DTSRequestRouting implements Serializable{
private String sourceID = null;private String sourceIDCode = null;private String recipientID = null;private String recipientIDCode = null;private String uuid = null;private String transmissionDateTime = null;
public String getSourceID() { return sourceID; }public void setSourceID(String newSourceID){
sourceID = newSourceID;}
public String getSourceIDCode() { return sourceIDCode; }
public void setSourceIDCode(String newSourceIDCode){
sourceIDCode = newSourceIDCode;}
public String getRecipientID() { return recipientID; }
public void setRecipientID(String newRecipientID){
recipientID = newRecipientID;}
public String getRecipientIDCode() { return recipientIDCode; }
public void setRecipientIDCode(String newRecipientIDCode){
recipientIDCode = newRecipientIDCode;}
public String getUUID() { return uuid; }
public void setUUID(String newUUID){
uuid = newUUID;}
public String getTransmissionDateTime(){
return transmissionDateTime;}
public void setTransmissionDateTime(String newTransDateTime){
transmissionDateTime = newTransDateTime;}
}
Java Creating the Container Classes for the Service (IN)
52
Java Creating the Container Classes for the Service (OUT)
package com.datatransportstandard.www.serializable;
import java.io.Serializable;
public class DTSResponseRouting implements Serializable{
private String sourceID = null;private String sourceIDCode = null;private String recipientID = null;private String recipientIDCode = null;private String uuid = null;private String transmissionDateTime = null;
public String getSourceID() { return sourceID; }public void setSourceID(String newSourceID){
sourceID = newSourceID;}
public String getSourceIDCode() { return sourceIDCode; }
public void setSourceIDCode(String newSourceIDCode){
sourceIDCode = newSourceIDCode;}
public String getRecipientID() { return recipientID; }
public void setRecipientID(String newRecipientID){
recipientID = newRecipientID;}
public String getRecipientIDCode() { return recipientIDCode; }
public void setRecipientIDCode(String newRecipientIDCode){
recipientIDCode = newRecipientIDCode;}
public String getUUID() { return uuid; }
public void setUUID(String newUUID){
uuid = newUUID;}
public String getTransmissionDateTime(){
return transmissionDateTime;}
public void setTransmissionDateTime(String newTransDateTime){
transmissionDateTime = newTransDateTime;}
}
53
Java Examples (Service) • Modify the WSDD <beanMapping
languageSpecificType="java:com.datatransportstandard.www.serializable.DTSRequestRouting" qname="ns2:DTSRequestRouting" xmlns:ns2="http://www.datatransport.com"/>
<beanMapping languageSpecificType="java:com.datatransportstandard.www.serializable.DTSResponseRouting" qname="ns3:DTSResponseRouting" xmlns:ns3="http://www.datatranportstandard.com"/>
• Augment the Code (inbound)
MessageContext ctx = MessageContext.getCurrentContext();
SOAPEnvelope requestEnv = ctx.getRequestMessage().getSOAPEnvelope();
SOAPHeaderElement requestHeader = requestEnv.getHeaderByName("http://www.datatransportstandard.com",
“DTSRequestRouting");DTSRequestRouting incontainer = (DTSRequestRouting) requestHeader.getObjectValue();System.out.println(“SourceId = “ + incontainer.getSourceId());
54
Java Examples (Service)
• Augment the Code (outbound)DTSResponseRouting outcontainer = new DTSResponseRouting();outcontainer.setSourceId(“TEST SOURCE ID”);outcontainer.setSourceIdType(“TEST SOURCE ID TYPE”);…
SOAPHeaderElement responseHeader = new SOAPHeaderElement(“http://www.datatransportstandard.com”,“DTSResponseRouting”)
responseHeader.setObjectValue(outcontainer);
SOAPEnvelope responseEnv = ctx.getResponseMessage().getSOAPEnvelope();
responseEnv.addHeader(responseHeader);
55
Java Examples (Client)
• Augment the Code (General)QName qn = new
QName(“http://www.datatransportstandard.com”,“DTSRequestRouting”);call.registerTypeMapping(DTSRequestRouting.class, qn,
new BeanSerializerFactory(DTSRequestRouting.class, qn),
new BeanDeserializerFactory(DTSRequestRouting.class, qn));
QName qn1 = new QName(“http://www.datatranpsortstandard.com”,“DTSResponseRouting”);call.registerTypeMapping(DTSResponseRouting.class, qn1,
new BeanSerializerFactory(DTSResponseRouting.class, qn1),
new BeanDeserializerFactory(DTSResponseRouting.class, qn1));
56
Java Examples (Client)
• Augment the Code (outbound)
DTSRequestRouting outcontainer = new DTSRequestRouting();outcontainer.setSourceId(“TEST SOURCE ID”);outcontainer.setSourceIdType(“TEST SOURCE ID TYPE”);…
SOAPHeaderElement requestHeader = new SOAPHeaderElement(“http://www.datatransportstandard.com”, “DTSRequestRouting”);
requestHeader.setObjectValue(outcontainer);
call.addHeader(requestHeader);
57
Java Examples (Client)
• Augment the Code (inbound)SOAPEnvelope responseEnv =
call.getMessageContext().getResponseMessage().getSOAPEnvelope();
SOAPHeaderElement responseHeader = responseEnv.getHeaderByName(“http://www.datatransportstandard.com”,“DTSResposneHeader”);
DTSResponseHeader incontainer = (DTSResponseHeader) responseHeader.getObjectValue();
System.out.println(“Source Id=“ + intcontainer.getSourceId());
58
.Net: Creating the Container classes for Service[XmlTypeAttribute(Namespace=“http://www.datatransportstandard.com”)][XmlRootAttribute(ElementName=“DTSRequestRouting”, Namespace=“http://www.datatransportstandard.com”, IsNullable=false)][XmlInclude(typeof(DTSRequestRouting))]public class DTSRequestRouting : System.Web.Services.Protocols.SoapHeader
{public string UUID;public string transmissionDateTime;public string sourceID;public string sourceIDCode;public string recipientID;
}
[XmlTypeAttribute(Namespace=“http://www.datatransportstandard.com”)][XmlRootAttribute(ElementName=“DTSResponseRouting”, Namespace=“http://www.datatransportstandard.com”, IsNullable=false)][XmlInclude(typeof(DTSResponseRouting))]public class DTSResponseRoutingElements : System.Web.Services.Protocols.SoapHeader
{ public string UUID;public string transmissionDateTime;public string sourceID;public string sourceIDCode;public string recipientID;
}[XmlTypeAttribute(Namespace=“http://www.datatransportstandard.com")]public class DTSResponseRouting : DTSResponseRoutingElements{};
59
.Net: Augment the Service
• Add declarations to service Classpublic DTSRequestRouting DTSRequestRoutingVal;
public DTSResponseRoutingElements DTSResponseRoutingVal;
public SoapUnknownHeader[] unknownHeaders;
• Add serialization directives to WebMethod()[SoapHeaderAttribute("DTSRequestRoutingVal")]
[SoapHeaderAttribute("DTSResponseRoutingVal", Direction=SoapHeaderDirection.Out)]
[SoapHeader("unknownHeaders")]
60
.Net: Creating the Container classes for Client[XmlTypeAttribute(Namespace="http://www.datatransportstandard.com")]
[XmlRootAttribute(ElementName="DTSRequestRouting",Namespace="http://www.datatransportstandard.com", IsNullable=false)]
[XmlInclude(typeof(DTSRequestRouting))]
public class DTSRequestRoutingElements : System.Web.Services.Protocols.SoapHeader
{
public string UUID;
public string transmissionDateTime;
public string sourceID;
public string sourceIDCode;
public string recipientID;
}
[XmlTypeAttribute(Namespace="http://www.datatransportstandard.com")]
public class DTSRequestRouting : DTSRequestRoutingElements{}
[XmlTypeAttribute(Namespace="http://www.datatransportstandard.com")]
[XmlIncludeAttribute(typeof(DTSResponseRouting))]
[XmlRootAttribute("DTSResponseRouting", Namespace="http://www.datatransportstandard.com", IsNullable=false)]
public class DTSResponseRouting : System.Web.Services.Protocols.SoapHeader
{
public string UUID;
public string transmissionDateTime;
public string sourceID;
public string sourceIDCode;
public string recipientID;
}
61
.Net: Augment the Client
• Add declarations to Client Web Reference/Proxy Class
public DTSRequestRoutingElements DTSRequestRoutingVal;
public DTSResponseRouting DTSResponseRoutingVal;
• Add serialization directives to WebMethod() [SoapHeaderAttribute("DTSRequestRoutingVal")]
[SoapHeaderAttribute("DTSResponseRoutingVal", Direction=SoapHeaderDirection.Out)]
62
We appreciate your feedback and comments. We can be reached at:
Nathan Chitty, Nelnet, Inc.(904) [email protected]
Gary Sandler, ELM Resources(510) [email protected]
Contact Information