External Interfaces & Integration July 24 2007 Stephen Kerr
description
Transcript of External Interfaces & Integration July 24 2007 Stephen Kerr
1 Integration Update
External Interfaces & Integration
July 24 2007
Stephen Kerr
2 Integration Update
Agenda
• Updates to External Interface Specification– New Services– Sandbox & EDS deployment– Notifications & Listeners & Security
• Web Service testing
• Integration Design Process– Artifacts– Process– Exceptions
3 Integration Update
Updates to External Interface Specification
• Certain elements of the External Interface Specification were approved on May 21st on condition that regular updates were provided to the specification
• Revision 1.02 was posted on July 20th for review– Review feedback required by August 3rd – Updated document will be republished by August 9th – TPTF vote on August 14th
• The changes in version 1.02 are– Removed deleteOutputSchedules from ThreePartOffer– Added statement related to ordering guarantees in section 2.6.– Added note regarding the handling of timestamps that are not interval ‘aligned’ in
section 2.3.2– Corrected figure 10 to reflect use of ‘INC’ or ‘DEC’ type in mRID for IncDecOffers– Corrected Figure 9 to not have expiration column– Modified Figure 10 to add column to map protocol terms to XML tags– Modified sections 3.3.1, 3.3.3, 3.3.4, 3.3.5 and 3.3.11 to not use expirationTime
as a key value– Corrected references to ‘IrregularIntervalSchedule’ to be ‘TmSchedule’– Revised section 5.1 and Figure 109 to reflect revised notification scheme– Revised ASOffer structure, uses a variation of PriceCurve that provides price
points for each specific AS type. This affects sections 3.3.4, 2.3.1, 2.3.3 and 2.3.4
4 Integration Update
New services being built
• The services we are building for Aug 30th release as loopback services are listed
• There will be no back-end functionality associated with these services in the Sandbox
• The back-end systems have not implemented these services yet and are subject to change
• The Market Information types will be simulated but we have not yet decided what the response will be– Simple document or XML response
• The services will be deployed to the EDS environment on the transition plan schedule
• New Market Transaction Services– Self Arranged Ancillary Services– Ancillary Services Offer– Ancillary Services Trade– DAM Energy Bid– DAM Energy Only Offer– Congestion Revenue Rights Offer– PTP Obligation Bid– Self Schedule
• New Market Information Services– Proxy Curves– Startup and Shutdown Instructions– Total Regulation– Load Ratio Share– Unit Availability– Forecasted Load– Real-Time System Load– Market LMPs and SPPs– Mitigated Curves
• New Notifications– Outage Notifications– Notices and Alerts– CRR Awards
• New Outage Scheduling Services– Outage Creation
5 Integration Update
Sandbox & EDS deployment
• Following services are available– Three Part Offer– Incremental/Decremental Offers– Current Operating Plan– Output Schedule– Bid Set Acceptance– Bid Set Errors– Get mRID– System Status
• Defects are being posted here:– http://nodal.ercot.com/readiness/sandbox/documents/docs/Sandbox_Defects.xls– Problem with optional fields being made mandatory
• Notifications system is functional but network configuration has to be reworked to get notifications through the DMZ
• Back-end integration with MMS enters test in early August for early EDS3– Three Part Offer– Incremental/ Decremental Offers– Current Operating Plan– Output Schedule
• Pending sent through the general Notification service NOT– Bid Set Acceptance– Bid Set Errors
6 Integration Update
Notifications & Listeners
Proxy Server
Request Broker
Notification Service
Event Source
Target Application
MP Application
HTTPS - Notify
HTTPS - Acknowledge
MP Service
HTTPS - Request
HTTPS - Reply
HTTPS
HTTPS
Event
Back-end systems will be available to stimulate notifications toward end
07/early 08.Until then notifications will be
simulated
Network design is in progress around the
outbound and inbound traffic through DMZ
7 Integration Update
Solution Use Case
The following sequence of steps would be used to send a notification message to an MP system:
1. ERCOT Notification Service establishes an HTTPS connection to target MP system at port 443 of URL pre-specified by MP
2. Notify message is signed by ERCOT using the ERCOT server certificate (the same one used for external web services)
3. MP web service is invoked by ERCOT Notification Service, sending the Notify message
4. MP system validates the signature, using the ERCOT public key5. Notify message is consumed by the MP system6. MP system replies with a simple, unsigned Acknowledge
message using the same HTTPS connection7. ERCOT Notification Service receives Acknowledge message and
marks the transmission as complete8. Process is repeated for next notification or if a retry is needed
according to established rules
8 Integration Update 8
Solution Approach
• MPs can use any server certificate of their choosing for HTTPS connections to their web server
• ERCOT will not use a client certificate when establishing an HTTPS connection to a MP notification listener service
• There is no need to manage additional client certificates over those needed for the external web services, greatly simplifying ERCOT implementation and administration
• MPs must be able to read ERCOT’s signature to verify that ERCOT sent the Notification
• MPs can not use Certificate Revocation Lists (CRLs), as ERCOT would not have a client certificate for each MP system
• Implementation is easier for MPs, as they don’t need to sign the Acknowledge messages (there is only non-sensitive data on the Acknowledge)
9 Integration Update 9
Steps to test Notifications & Listeners
• MP’s must get a new Nodal Test Certificate from ERCOT. The detailed instructions for this request will be communicated through the EDS meetings
• MP’s must get ERCOT’s public certificate from their Client Services representative in order to read ERCOT signatures
• MP’s must get the WSDL & XSD from ERCOT, published on http://nodal.ercot.com/readiness/sandbox/websvc/index.html
• MP’s are responsible for constructing the Notification listener service, using whatever technologies suits
• MP’s must notify ERCOT of the listener addresses (primary & secondary) and email details (via [email protected])
• MP’s must set up their infrastructure to accept an https connection from ERCOT
• MP’s can schedule time with ERCOT to test out the service – we will provide 1:1 time with developers (via [email protected])
• ERCOT verifies the service by sending a test Notification to the MP’s service
10 Integration Update
Testing for EWS Loopback release
• Testing was done a three levels
– Development• Developers executed 19 formal unit tests, along with a large number of informal tests
• Issue defects were fixed
– iFAT• iFAT test team executed 133 formal test scripts
– Common services (audit and error handling) 5 tests
– External clients 11 tests
– Functional 30 tests
– Notifications 10 tests
– Security 5 tests
– XML validation 72 tests
• Identified 36 issues or defects
• All issues were resolved
– Smoke test of Sandbox• iFAT test team executed 14 formal test scripts
• Found that notifications fail because of the setup of the sandbox environment
11 Integration Update
Sample test case
Step Name
Description Expected
Step 1 Pick the attached sample 3PO valid xml Sample xml is successfully picked.
Step 2 set the following Changes are reflected.
verb = change
Set FipFop value to 850 or make any other changes to payload/bidset
Step 3 Place this xml in the folder used by test harness client Xml is placed in the folder.
Step 4 A request is to be sent and a descriptive message is expected along with Ok Ok is expected.
Step 5 Do a "Get" request depending on the mrID from the step 4. A response reflecting the changes should be displayed
Step 6 Validate that change is reflected in the response message. Change should be visible.
Step 7 Once the test is complete set the xml back to its sample state. Xml is reset.
12 Integration Update
Interface Design Process
• Identify Integrations– System of Systems Architecture (SoSA) describes the enterprise level integration points
needed for the overall Nodal solution.– Integration Design Authority (Technical & Business Architects), project teams and
integration vendors work together to produce artifacts
• Project Teams are accountable for determining what information they need – Project teams (CRR, MMS, CMM etc) create CSDs & Use Cases that indicate dependencies
on information from other systems– Project teams determine what information is necessary and whether they can be supplied– Integration picks up from that point
• Interface Specifications & Design– Integration artifacts are produced that contain:
• Data Element Name, Element Attributes, Volumetrics, Data Frequency etc.• Business Triggering Events that drive the movement of information
– The integration teams use these artifacts to analyze the interface and develop Use Cases• Identify gaps, integration characteristics, mismatches and transformations of data that is necessary between systems
– Integrations specifications and Use Cases are the input to design and implementation of the integration between systems.
• Exceptions are handled through IDA and Design ClearingHouse– Cases where projects have made conflicting assumptions – Cases where integration discovers missing or incorrect information elements that do not
match
13 Integration Update
The integration factory constructs interfaces
Interface Build Plan
Interface Specification
Interface Design
Interface Build
FAT & ITESTInterface
teamassigned
Interface specs agreed
Interface design agreed
Interface conforms with spec
Inputs
Activities
EDS ScheduleEDS DefinitionsProject SchedulesInterface Priorities
Seek project commitments
Ramp up design and build teams
Produce iteration plan for the interface
Outline the information flows in the interface
Information flows in the interface are enumerated
Interface team resolves discrepancies in interface definition between source, target & system
Define behavior and information contained in the interface
SoSA – use cases & domainProject CSDs & UCsInterface docs from source and target systems
Integration design patternsExisting designsSource & target designs (CSD & Detailed Design)
Numerous build inputs incl. Test Cases, Environment config, makefiles, …
Integration pattern selected
Impacts on source and target assessed and incorporated
Collaborate with Source & target systems teams
Spec stubs and data
Interface tests defined
Updated design documentation
Updated test and dev environments
Updated source and libraries
Stubs built
14 Integration Update
Artifacts from the integration
Registration Interface Spec
Sample artifacts that are in progress
S&B - CMM Spec
Designs are based on our pattern library:
15 Integration Update
Artifacts from the integration
Designs are based on our pattern library:
JMS Consumer Pattern