PC 901 AdvWorkflowGuide En
Transcript of PC 901 AdvWorkflowGuide En
-
7/27/2019 PC 901 AdvWorkflowGuide En
1/307
Informatica PowerCenter (Version 9.0.1)
Advanced Workflow Guide
-
7/27/2019 PC 901 AdvWorkflowGuide En
2/307
Informatica PowerCenter Advanced Workflow Guide
Version 9.0.1June 2010
Copyright (c) 1998-2010 Informatica. All rights reserved.
This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use anddisclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any forby any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S. and/or internatioPatents and other Patents Pending.
Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided iDFARS 227.7202-1(a) and 227.7702-3(a) (1995), DFARS 252.227-7013(1)(ii) (OCT 1988), FAR 12.212(a) (1995), FAR 52.227-19, or FAR 52.227-14 (ALT III), as applica
The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us inwriting.
Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange,PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange and InformaticOn Demand are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and produnames may be trade names or trademarks of their respective owners.
Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rightsreserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rightsreserved.Copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright 2007 Isomorphic Software. All rights reserved. Copyright MeIntegration Technology, Inc. All rights reserved. Copyright Intalio. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. Arights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved.Copyright Rouge Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright
Glyph & Cog, LLC. All rights reserved.
This product includes software developed by the Apache Software Foundation (http://www.apache.org/), and other software which is licensed under the Apache License,Version 2.0 (the "License"). You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writingsoftware distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLicense for the specific language governing permissions and limitations under the License.
This product includes software which was developed by Mozilla (http://www.mozilla.org/), software copyright The JBoss Group, LLC, all rights reserved; software copyright1999-2006 by Bruno Lowagie and Paulo Soares and other software which is licensed under the GNU Lesser General Public License Agreement, which may be found at httwww.gnu.org/licenses/lgpl.html. The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but notlimited to the implied warranties of merchantability and fitness for a particular purpose.
The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvineand Vanderbilt University, Copyright () 1993-2006, all rights reserved.
This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistributionthis software is subject to terms available at http://www.openssl.org.
This product includes Curl software which is Copyright 1996-2007, Daniel Stenberg, . All Rights Reserved. Permissions and limitations regarding thissoftware are subject to terms available at http://curl.haxx.se/docs/copyright.html. Permission to use, copy, modify, and distribute this software for any purpose with or withoufee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
The product includes software copyright 2001-2005 () MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms availabat http://www.dom4j.org/ license.html.
The product includes software copyright 2004-2007, The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to termsavailable at http:// svn.dojotoolkit.org/dojo/trunk/LICENSE.
This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding thsoftware are subject to terms available at http://source.icu-project.org/repos/icu/icu/trunk/license.html.
This product includes software copyright 1996-2006 Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at httpwww.gnu.org/software/ kawa/Software-License.html.
This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & WirelessDeutschland. Permissions and limitations regarding this software are subject to terms available at http://www.opensource.org/licenses/mit-license.php.
This product includes software developed by Boost (http://www.boost.org/) or under the Boost software license. Permissions and limitations regarding this software are subjto terms available at http:/ /www.boost.org/LICENSE_1_0.txt.
This product includes software copyright 1997-2007 University of Cambridge. Permissions and limitations regarding this software are subject to terms available at http://www.pcre.org/license.txt.
This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to termsavailable at http:// www.eclipse.org/org/documents/epl-v10.php.
This product includes software li censed under the terms at http://www.tcl.tk/software/tcltk/license.html, http://www.bosrup.com/web/overlib/?License, http://www.stlport.org/dlicense.html, http://www.asm.ow2.org/license.html, http://www.cryptix.org/LICENSE.TXT, http://hsqldb.org/web/hsqlLicense.html, http://httpunit.sourceforge.net/doc/license.html, http://jung.sourceforge.net/license.txt , http://www.gzip.org/zlib/zlib_license.html, http://www.openldap.org/software/release/license.html, http://www.libssh2.org
http://slf4j.org/license.html, http://www.sente.ch/software/OpenSourceLicense.html, and http://fusesource.com/downloads/license-agreements/fuse-message-broker-v-5-3-license-agreement.
This product includes software licensed under the Academic Free License (http://www.opensource.org/licenses/afl-3.0.php), the Common Development and DistributionLicense (http://www.opensource.org/licenses/cddl1.php) the Common Public License (http://www.opensource.org/licenses/cpl1.0.php) and the BSD License (http://www.opensource.org/licenses/bsd-license.php).
This product includes software copyright 2003-2006 Joe WaInes, 2006-2007 XStream Committers. All rights reserved. Permissions and limitations regarding this softwareare subject to terms available at http://xstream.codehaus.org/license.html. This product includes software developed by the Indiana University Extreme! Lab. For furtherinformation please visit http://www.extreme.indiana.edu/.
This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,501; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775;6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,162,643; 7,254,590; 7,281,001; 7,421,458; and 7,584,422, international Patents and otherPatents Pending..
-
7/27/2019 PC 901 AdvWorkflowGuide En
3/307
DISCLAIMER: Informatica Corporation provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the impliedwarranties of non-infringement, merchantability, or use for a particular purpose. Informatica Corporation does not warrant that this software or documentation is error free. Tinformation provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation issubject to change at any time without notice.
NOTICES
This Informatica product (the Software) includes certain drivers (the DataDirect Drivers) from DataDirect Technologies, an operating company of Progress SoftwareCorporation (DataDirect) which are subject to the following terms and conditions:
1. THE DATADIRECT DRIVERS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.
2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF
THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACHOF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS.
Part Number: PC-AWG-90100-0001
-
7/27/2019 PC 901 AdvWorkflowGuide En
4/307
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica Customer Portal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica Web Site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica How-To Library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Informatica Knowledge Base. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Informatica Multimedia Knowledge Base. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Informatica Global Customer Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Chapter 1: Understanding Pipeline Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Understanding Pipeline Partitioning Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Partitioning Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Partition Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Numberof Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Partition Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Dynamic Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Configuring Dynamic Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Rules and Guidelines for Dynamic Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Using Dynamic Partitioning with Partition Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Configuring Partition-Level Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Cache Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Mapping Variables in Partitioned Pipelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Partitioning Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Partition Restrictions for Editing Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Partition Restrictions for PowerExchange. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Configuring Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Adding Partition Points to a Pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Configuring a Partition Point. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Partition Points Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Non-Partition Points Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Chapter 2: Partition Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Partition Points Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Adding and Deleting Partition Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Rules and Guidelines for Adding and Deleting Partition Points. . . . . . . . . . . . . . . . . . . . . . . . 13
Partitioning Relational Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Entering an SQL Query. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Table of Contents i
-
7/27/2019 PC 901 AdvWorkflowGuide En
5/307
Entering a Filter Condition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Partitioning File Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Rules and Guidelines for Partitioning File Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Using One Thread to Read a File Source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Using Multiple Threads to Read a File Source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Configuring for File Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Partitioning Relational Targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Database Compatibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Partitioning File Targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Configuring Connection Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Configuring File Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Partitioning Custom Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Working with Multiple Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Creating Partition Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Working with Threads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Partitioning Joiner Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Partitioning Sorted Joiner Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Using Sorted Flat Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Using Sorted Relational Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Using SorterTransformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Optimizing Sorted Joiner Transformations with Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Partitioning Lookup Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Cache Partitioning Lookup Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Partitioning Pipeline Lookup Transformation Cache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Partitioning Sequence Generator Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Partitioning SorterTransformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Configuring Sorter Transformation Work Directories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Partitioning XML Generator Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Restrictions for Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Restrictions for Numerical Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Chapter 3: Partition Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Partition Types Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Setting Partition Types in the Pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Setting Partition Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Database Partitioning Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Partitioning Database Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Target Database Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Hash Auto-Keys Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Hash User Keys Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Key Range Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Adding a Part ition Key. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Adding Key Ranges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
ii Table of Contents
-
7/27/2019 PC 901 AdvWorkflowGuide En
6/307
Pass-Through Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Round-Robin Partition Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter 4: Pushdown Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Pushdown Optimization Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Pushdown Optimization Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Running Source-Side Pushdown Optimization Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Running Target-Side Pushdown Optimization Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Running Full Pushdown Optimization Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Active and Idle Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Working with Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Comparing the Output of the Integration Service and Databases. . . . . . . . . . . . . . . . . . . . . . 50
Using ODBC Drivers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Rules and Guidelines for Netezza. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Pushdown Compatibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Incompatible Users for Database Connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Qualifying Names of Tables in Idle Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Working with Dates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Working with Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Rules and Guidelines for Functions in Pushdown Optimization. . . . . . . . . . . . . . . . . . . . . . . . 60
Error Handling, Logging, and Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Error Handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Working with Slowly Changing Dimensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Working with Sequences and Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Troubleshooting Orphaned Sequences and Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Using the $$PushdownConfig Mapping Parameter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Configuring Sessions for Pushdown Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Pushdown Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Target Load Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Viewing Pushdown Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Chapter 5: Pushdown Optimization Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Pushdown Optimization and Transformations Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Rules and Guidelines for Pushdown Optimization and Transformations. . . . . . . . . . . . . . . . . . 74
Aggregator Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Table of Contents iii
-
7/27/2019 PC 901 AdvWorkflowGuide En
7/307
Expression Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Filter Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Joiner Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Lookup Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Unconnected Lookup Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Lookup Transformation with an SQL Override. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Router Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Sequence Generator Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Sorter Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Source Qualifier Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Source Qualifier Transformation with an SQL Override. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Union Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Update Strategy Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Chapter 6: Real-time Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Real-time Processing Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Understanding Real-time Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Messages and Message Queues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Web Service Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Change Data from PowerExchange Change Data Capture Sources. . . . . . . . . . . . . . . . . . . . 88
Configuring Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Terminating Conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Idle Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Message Count. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Reader Time Limit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Flush Latency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Commit Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Steps to Enable Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Recovery File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Message Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Session Recovery Data Flush. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Recovery Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
PM_REC_STATE Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Message Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Recovery Queue and Recovery Topic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Message Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Recovery Ignore List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
iv Table of Contents
-
7/27/2019 PC 901 AdvWorkflowGuide En
8/307
Stopping Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Restarting and Recovering Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Restarting Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Recovering Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Restart and Recover Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Rules and Guidelines for Real-time Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Rules and Guidelines for Message Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Real-time Processing Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Informatica Real-time Products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100
Chapter 7: Commit Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Commit Points Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Target-Based Commits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Source-Based Commits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102
Determining the Commit Source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Switching from Source-Based to Target-Based Commit. . . . . . . . . . . . . . . . . . . . . . . . . . . 104
User-Defined Commits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Rolling Back Transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Understanding Transaction Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Transformation Scope. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .110
Understanding Transaction Control Units. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112
Rules and Guidelines for Working with Transaction Control. . . . . . . . . . . . . . . . . . . . . . . . .112
Creating Target Files by Transaction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Setting Commit Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Chapter 8: Row Error Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Row Error Logging Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115Error Log Code Pages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116
Understanding the Error Log Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116
PMERR_DATA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
PMERR_MSG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118
PMERR_SESS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
PMERR_TRANS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Understanding the Error Log File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121
Configuring Error Log Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123
Chapter 9: Workflow Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125Workflow Recovery Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
State of Operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Workflow State of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Session State of Operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126
Target Recovery Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127
Recovery Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .129
Table of Contents v
-
7/27/2019 PC 901 AdvWorkflowGuide En
9/307
Suspending the Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130
Configuring Suspension Email. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Configuring Workflow Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Recovering Stopped, Aborted, and Terminated Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . 132
Recovering Suspended Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Configuring Task Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Task Recovery Strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Automatically Recovering Terminated Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Resuming Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Working with Repeatable Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Source Repeatability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Transformation Repeatability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Configuring a Mapping for Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Steps to Recover Workflows and Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Recovering a Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Recovering a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Recovering a Workflow From a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141Rules and Guidelines for Session Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Configuring Recovery to Resume from the Last Checkpoint. . . . . . . . . . . . . . . . . . . . . . . . . 141
Unrecoverable Workflows or Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Chapter 10: Stopping and Aborting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Stopping andAborting Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Types of Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Threshold Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Fatal Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Integration Service Handling for Session Failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145Stopping or Aborting the Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Stopping or Aborting a Task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Steps to Stop or Abort. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Chapter 11: Con c u r r e n t W o r k f l o w s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 4 7
Concurrent Workflows Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Configuring Unique Workflow Instances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Recovering Workflow Instances by Instance Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Rules and Guidelines for Running Concurrent Instances of the Same Instance Name. . . . . . . . 148
Configuring Concurrent Workflows of the Same Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Running Concurrent Web Service Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Configuring Workflow Instances of the Same Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Recovering Workflow Instances of the Same Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Rules and Guidelines for Running Concurrent Instances of the Same Instance Name. . . . . . . . 150
Using Parameters and Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Accessing the Run Instance Name or Run ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
vi Table of Contents
-
7/27/2019 PC 901 AdvWorkflowGuide En
10/307
Steps to Configure Concurrent Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Starting and Stopping Concurrent Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Starting Workflow Instances from Workflow Designer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Starting One Concurrent Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Starting Concurrent Workflows from the Command Line. . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Stopping or Aborting Concurrent Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Monitoring Concurrent Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Viewing Session and Workflow Logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Log Files for Unique Workflow Instances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Log Files forWorkflow Instances of the Same Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Rules and Guidelines for Concurrent Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Chapter 12: Grid Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Grid Processing Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Running Workflows on a Grid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Running Sessions on a Grid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Working with Partition Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Forming Partition Groups Without Resource Requirements. . . . . . . . . . . . . . . . . . . . . . . . . 158
Forming Partition Groups With Resource Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Rules and Guidelines for Creating Partition Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Working with Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Grid Connectivity and Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Configuring a Workflow or Session to Run on a Grid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Rules and Guidelines for Configuring a Workflow or Session to Run on a Grid. . . . . . . . . . . . . 160
Chapter 13: Load Balancer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Load Balancer Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161Assigning Service Levels to Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Assigning Resources to Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Chapter 14: W o r k f l o w V a r i a b l e s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 6 4
Workflow Variables Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Predefined Workflow Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Using Predefined Workflow Variables in Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Evaluating Condition in a Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Evaluating Task Status in a Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Evaluating Previous Task Status in a Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168User-Defined Workflow Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Workflow Variable Start and Current Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Datatype Default Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Creating User-Defined Workflow Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Using Worklet Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Persistent Worklet Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Table of Contents vii
-
7/27/2019 PC 901 AdvWorkflowGuide En
11/307
Overriding the Initial Value. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Rules and Guidelines for Using Worklet Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Assigning Variable Values in a Worklet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Passing Variable Values between Worklets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Configuring Variable Assignments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Chapter 15: Parameters and Variables in Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Working with Session Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Changing the Session Log Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Changing the Target File and Directory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Changing Source Parameters in a File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Changing Connection Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Getting Run-Time Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Rules and Guidelines for Creating File Parameters and Database Connection Parameters. . . . . 180
Mapping Parameters and Variables in Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Assigning Parameter and Variable Values in a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Passing Parameter and Variable Values between Sessions. . . . . . . . . . . . . . . . . . . . . . . . . 182
Configuring Variable Assignments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Chapter 16: Parameter Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Parameter Files Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Parameter and Variable Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Where to Use Parameters and Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Overriding Connection Attributes in the Parameter File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Parameter File Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Parameter File Sections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196Null Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Sample Parameter File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Configuring the Parameter File Name and Location. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Using a Parameter File with Workflows or Sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Using a Parameter File with pmcmd. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Parameter File Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Guidelines for Creating Parameter Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Troubleshooting Parameters and Parameter Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Tips for Parameters and Parameter Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Chapter 17: FastExport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Using FastExport Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Step 1. Create a FastExport Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Verifying the Code Page Mapping File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Step 2. Change the Reader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Step 3. Change the Source Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
viii Table of Contents
-
7/27/2019 PC 901 AdvWorkflowGuide En
12/307
Step 4. Override the Control File (Optional). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Rules and Guidelines for Using FastExport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Chapter 18: Ext e r n a l L o a d i n g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 0 8
External Loading Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
External Loader Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Loading Data to a Named Pipe. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Staging Data to a Flat File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Partitioning Sessions with External Loaders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Loading to IBM DB2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
IBM DB2 EE External Loader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
IBM DB2 EEE External Loader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Rules and Guidelines for IBM DB2 EEE External Loaders. . . . . . . . . . . . . . . . . . . . . . . . . . 211
Setting Operation Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Configuring Authorities, Privileges, and Permissions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Configuring IBM DB2 EE External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Configuring IBM DB2 EEE External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Loading to Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Rules and Guidelines for Oracle External Loaders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Loading Multibyte Data to Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Configuring Oracle External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Loading to Sybase IQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Rules and Guidelines for Sybase IQ External Loaders. . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Loading Multibyte Data to Sybase IQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Configuring Sybase IQ External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Loading to Teradata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219Rules and Guidelines for Teradata External Loaders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Overriding the Control File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Creating UserVariables in the Control File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Configuring Teradata MultiLoad External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . 222
Configuring Teradata TPump External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Configuring Teradata FastLoad External Loader Attributes. . . . . . . . . . . . . . . . . . . . . . . . . 226
Configuring Teradata Warehouse Builder Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Configuring External Loading in a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Configuring a Session to Write to a File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Configuring File Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Selecting an External Loader Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Troubleshooting External Loading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Chapter 19: FTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
FTP Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Rules and Guidelines for Using FTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Table of Contents ix
-
7/27/2019 PC 901 AdvWorkflowGuide En
13/307
Integration Service Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Using FTP with Source Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .236
Using FTP with Target Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Configuring FTP in a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Configuring SFTP in a Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Selecting an FTP Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Configuring Source File Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .238
Configuring Target File Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Chapter 20: Session Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Session Caches Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Cache Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Cache Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Naming Convention for Cache Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .243
Cache File Directory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .245
Configuring the Cache Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Calculating the Cache Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .246
Using Auto Memory Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Configuring a Numeric Cache Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .247
Steps to Configure the Cache Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .247
Cache Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Configuring the Cache Size for Cache Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .248
Aggregator Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .249
Incremental Aggregation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Configuring the Cache Sizes for an Aggregator Transformation. . . . . . . . . . . . . . . . . . . . . . 249
Troubleshooting Aggregator Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .250
Joiner Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2501:n Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
n:n Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Configuring the Cache Sizes for a Joiner Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . 252
Troubleshooting Joiner Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252
Lookup Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Sharing Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .254
Configuring the Cache Sizes for a Lookup Transformation. . . . . . . . . . . . . . . . . . . . . . . . . .254
Rank Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Configuring the Cache Sizes for a Rank Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . .255
Troubleshooting Rank Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
Sorter Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .256
Configuring the Cache Size for a Sorter Transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . 257
XML Target Caches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Configuring the Cache Size for an XML Target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .257
Optimizing the Cache Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .258
x Table of Contents
-
7/27/2019 PC 901 AdvWorkflowGuide En
14/307
Chapter 21: I ncremental A ggregation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .259
Incremental Aggregation Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Integration Service Processing for Incremental Aggregation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Reinitializing the Aggregate Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Moving or Deleting the Aggregate Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Finding Index and Data Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Partitioning Guidelines with Incremental Aggregation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Preparing forIncremental Aggregation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Configuring the Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Configuring the Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Chapter 22: Session Log Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Session Log Interface Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Implementing the Session Log Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
The Integration Service and the Session Log Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Rules and Guidelines for Implementing the Session Log Interface. . . . . . . . . . . . . . . . . . . . . 265
Functions in the Session Log Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
INFA_InitSessionLog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
INFA_OutputSessionLogMsg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
INFA_OutputSessionLogFatalMsg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
INFA_EndSessionLog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
INFA_AbnormalSessionTermination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Session Log Interface Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Building the External Session Log Library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Using the External Session Log Library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Chapter 23: Understanding Buffer Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Understanding Buffer Memory Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Configuring Automatic Memory Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Configuring Automatic Memory Settings for Multiple Sessions. . . . . . . . . . . . . . . . . . . . . . . 271
Configuring Buffer Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Configuring Session Cache Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Configuring Maximum Memory Limits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Configuring Automatic Memory Settings for Session Caches. . . . . . . . . . . . . . . . . . . . . . . . 273
Chapter 24: High Precision Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275High Precision Data Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Bigint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Decimal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Table of Contents xi
-
7/27/2019 PC 901 AdvWorkflowGuide En
15/307
Preface
The PowerCenter Advanced Workflow Guide is written for developers and administrators who are responsible for
creating workflows and sessions, and running workflows. This guide assumes you have knowledge of your
operating systems, relational database concepts, and the database engines, flat files or mainframe system in your
environment. This guide also assumes you are familiar with the interface requirements for your supporting
applications.
Informatica Resources
Informatica Customer Portal
As an Informatica customer, you can access the Informatica Customer Portal si te at
http://mysupport.informatica.com. The site contains product information, user group information, newsletters,
access to the Informatica customer support case management system (ATLAS), the Informatica How-To Library,
the Informatica Knowledge Base, the Informatica Multimedia Knowledge Base, Informatica Product
Documentation, and access to the Informatica user community.
Informatica DocumentationThe Informatica Documentation team takes every effort to create accurate, usable documentation. If you have
questions, comments, or ideas about this documentation, contact the Informatica Documentation team through
email at [email protected]. We will use your feedback to improve our documentation. Let us
know if we can contact you regarding your comments.
The Documentation team updates documentation as needed. To get the latest documentation for your product,
navigate to Product Documentation from http://mysupport.informatica.com.
Informatica Web Site
You can access the Informatica corporate web site at http://www.informatica.com. The site contains information
about Informatica, its background, upcoming events, and sales offices. You will also find product and partnerinformation. The services area of the site includes important information about technical support, training and
education, and implementation services.
Informatica How-To Library
As an Informatica customer, you can access the Informatica How-To Library at http:/ /mysupport.informatica.com.
The How-To Library is a collection of resources to help you learn more about Informatica products and features. It
xii
http://www.informatica.com/mailto:[email protected]://mysupport.informatica.com/http://mysupport.informatica.com/http://www.informatica.com/http://mysupport.informatica.com/mailto:[email protected]://mysupport.informatica.com/ -
7/27/2019 PC 901 AdvWorkflowGuide En
16/307
includes articles and interactive demonstrations that provide solutions to common problems, compare features and
behaviors, and guide you through performing specific real-world tasks.
Informatica Knowledge Base
As an Informatica customer, you can access the Informatica Knowledge Base at http: //mysupport.informatica.com.
Use the Knowledge Base to search for documented solutions to known technical issues about Informatica
products. You can also find answers to frequently asked questions, technical white papers, and technical tips. If
you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base
team through email at [email protected].
Informatica Multimedia Knowledge Base
As an Informatica customer, you can access the Informatica Multimedia Knowledge Base at
http://mysupport.informatica.com. The Multimedia Knowledge Base is a collection of instructional multimedia files
that help you learn about common concepts and guide you through performing specific tasks. If you have
questions, comments, or ideas about the Multimedia Knowledge Base, contact the Informatica Knowledge Base
team through email at [email protected].
Informatica Global Customer Support
You can contact a Customer Support Center by telephone or through the Online Support. Online Support requires
a user name and password. You can request a user name and password at http://mysupport.informatica.com.
Use the following telephone numbers to contact Informatica Global Customer Support:
North America / South America Europe / Middle East / Africa Asia / Australia
Toll Free
+1 877 463 2435
Standard RateBrazil: +55 11 3523 7761Mexico: +52 55 1168 9763
United States: +1 650 385 5800
Toll Free
00 800 4632 4357
Standard RateBelgium: +32 15 281 702France: +33 1 41 38 92 26
Germany: +49 1805 702 702Netherlands: +31 306 022 797Spain and Portugal: +34 93 480 3760
United Kingdom: +44 1628 511 445
Toll Free
Austra lia: 1 800 151 830Singapore: 001 800 4632 4357
Standard RateIndia: +91 80 4112 5738
Preface xiii
http://mysupport.informatica.com/mailto:[email protected]://mysupport.informatica.com/mailto:[email protected]://mysupport.informatica.com/ -
7/27/2019 PC 901 AdvWorkflowGuide En
17/307
xiv
-
7/27/2019 PC 901 AdvWorkflowGuide En
18/307
C H A P T E R 1
Understanding Pipeline Partitioning
This chapter includes the following topics:
Understanding Pipeline Partitioning Overview, 1
Partitioning Attributes, 2
Dynamic Partitioning, 4
Cache Partitioning, 6
Mapping Variables in Partitioned Pipelines, 7
Partitioning Rules, 8
Configuring Partitioning, 9
Understanding Pipeline Partitioning Overview
You create a session for each mapping you want the Integration Service to run. Each mapping contains one or
more pipelines. A pipeline consists of a source qualifier and all the transformations and targets that receive data
from that source qualifier. When the Integration Service runs the session, it can achieve higher performance by
partitioning the pipeline and performing the extract, transformation, and load for each partition in parallel.
A parti tion is a pipel ine stage that executes in a single reader, transformation, or wri ter thread. The number of
partitions in any pipeline stage equals the number of threads in the stage. By default, the Integration Service
creates one partition in every pipeline stage.
If you have the Partitioning option, you can configure multiple partitions for a single pipeline stage. You can
configure partitioning information that controls the number of reader, transformation, and writer threads that the
master thread creates for the pipeline. You can configure how the Integration Service reads data from the source,
distributes rows of data to each transformation, and writes data to the target. You can configure the number of
source and target connections to use.
Complete the following tasks to configure partitions for a session:
Set partition attributes including partition points, the number of partitions, and the partition types.
You can enable the Integration Service to set partitioning at run time. When you enable dynamic partitioning,the Integration Service scales the number of session partitions based on factors such as the source database
partitions or the number of nodes in a grid.
After you conf igure a session for partit ioning, you can configure memory requirements and cache directories for
each transformation.
The Integration Service evaluates mapping variables for each partition in a target load order group. You can
use variable functions in the mapping to set the variable values.
1
-
7/27/2019 PC 901 AdvWorkflowGuide En
19/307
When you create multiple partitions in a pipeline, the Workflow Manager verifies that the Integration Service
can maintain data consistency in the session using the partitions. When you edit object properties in the
session, you can impact partitioning and cause a session to fail.
You add or edit partition points in the session properties. When you change partition points you can define the
partition type and add or delete partitions.
Partitioning Attributes
You can set the following attributes to partition a pipeline:
Partition points. Partition points mark thread boundaries and divide the pipeline into stages. The Integration
Service redistributes rows of data at partition points.
Number of partitions. A partition is a pipeline stage that executes in a single thread. If you purchase the
Partitioning option, you can set the number of partitions at any partition point. When you add partitions, you
increase the number of processing threads, which can improve session performance.
Partition types. The Integration Service creates a default partition type at each partition point. If you have thePartitioning option, you can change the partition type. The partition type controls how the Integration Service
distributes data among partitions at partition points.
Partition Points
By default, the Integration Service sets partition points at various transformations in the pipeline. Partition points
mark thread boundaries and divide the pipeline into stages. A stage is a section of a pipeline between any two
partition points. When you set a partition point at a transformation, the new pipeline stage includes that
transformation.
The following figure shows the default partition points and pipeline stages for a mapping with one pipeline:
When you add a partition point, you increase the number of pipeline stages by one. Similarly, when you delete a
partition point, you reduce the number of stages by one. Partition points mark the points in the pipeline where the
Integration Service can redistribute data across partitions.
For example, if you place a partition point at a Filter transformation and define multiple partitions, the IntegrationService can redistribute rows of data among the partitions before the Filter transformation processes the data. The
partition type you set at this partition point controls the way in which the Integration Service passes rows of data to
each partition.
2 Chapter 1: Understanding Pipeline Partitioning
-
7/27/2019 PC 901 AdvWorkflowGuide En
20/307
RELATED TOPICS:
Partition Points on page 12
Number of Partitions
The number of threads that process each pipeline stage depends on the number of partitions. A partition is apipeline stage that executes in a single reader, transformation, or writer thread. The number of partitions in any
pipeline stage equals the number of threads in that stage.
You can define up to 64 partitions at any partition point in a pipeline. When you increase or decrease the number
of partitions at any partition point, the Workflow Manager increases or decreases the number of partitions at all
partition points in the pipeline. The number of partitions remains consistent throughout the pipeline. If you define
three partitions at any partition point, the Workflow Manager creates three partitions at all other partition points in
the pipeline. In certain circumstances, the number of partitions in the pipeline must be set to one.
Increasing the number of partitions or partition points increases the number of threads. Therefore, increasing the
number of partitions or partition points also increases the load on the node. If the node contains enough CPU
bandwidth, processing rows of data in a session concurrently can increase session performance. However, if you
create a large number of partitions or partition points in a session that processes large amounts of data, you can
overload the system.
The number of partitions you create equals the number of connections to the source or target. If the pipeline
contains a relational source or target, the number of partitions at the source qualifier or target instance equals the
number of connections to the database. If the pipeline contains file sources, you can configure the session to read
the source with one thread or with multiple threads.
For example, when you define three partitions across the mapping, the master thread creates three threads at
each pipeline stage, for a total of 12 threads.
The Integration Service runs the partition threads concurrently. When you run a session with multiple partitions,
the threads run as follows:
1. The reader threads run concurrently to extract data from the source.
2. The transformation threads run concurrently in each transformation stage to process data. The Integration
Service redistributes data among the partitions at each partition point.
3. The writer threads run concurrently to write data to the target.
Partitioning Multiple Input Group Transformations
The master thread creates a reader and transformation thread for each pipeline in the target load order group. A
target load order group has multiple pipelines when it contains a transformation with multiple input groups.
When you connect more than one pipeline to a multiple input group transformation, the Integration Service
maintains the transformation threads or creates a new transformation thread depending on whether or not the
multiple input group transformation is a partition point:
Partition point does not exist at multiple input group transformation. When a partition point does not exist
at a multiple input group transformation, the Integration Service processes one thread at a time for the multiple
input group transformation and all downstream transformations in the stage.
Partition point exists at multiple input group transformation. When a partition point exists at a multiple
input group transformation, the Integration Service creates a new pipeline stage and processes the stage with
one thread for each partition. The Integration Service creates one transformation thread for each partition
regardless of the number of output groups the transformation contains.
Partitioning Attributes 3
-
7/27/2019 PC 901 AdvWorkflowGuide En
21/307
Partition Types
When you configure the partitioning information for a pipeline, you must define a partition type at each partition
point in the pipeline. The partition type determines how the Integration Service redistributes data across part ition
points.
The Integration Services creates a default partition type at each partition point. If you have the Partitioning option,you can change the partition type. The partition type controls how the Integration Service distributes data among
partitions at partition points. You can create different partition types at different points in the pipeline.
You can define the following partition types in the Workflow Manager:
Database partitioning. The Integration Service queries the IBM DB2 or Oracle database system for table
partition information. It reads partitioned data from the corresponding nodes in the database. You can use
database partitioning with Oracle or IBM DB2 source instances on a multi-node tablespace. You can use
database partitioning with DB2 targets.
Hash auto-keys. The Integration Service uses a hash function to group rows of data among partitions. The
Integration Service groups the data based on a partition key. The Integration Service uses all grouped or sorted
ports as a compound partition key. You may need to use hash auto-keys partitioning at Rank, Sorter, and
unsorted Aggregator transformations.
Hash user keys. The Integration Service uses a hash function to group rows of data among partitions. Youdefine the number of ports to generate the partition key.
Key range. With key range partitioning, the Integration Service distributes rows of data based on a port or set
of ports that you define as the par tition key. For each port, you define a range of values. The Integration
Service uses the key and ranges to send rows to the appropriate partition. Use key range partitioning when the
sources or targets in the pipeline are partitioned by key range.
Pass-through. In pass-through partitioning, the Integration Service processes data without redistributing rows
among partitions. All rows in a single partition stay in the par tition after crossing a pass-through partition point.
Choose pass-through partitioning when you want to create an additional pipeline stage to improve
performance, but do not want to change the distribution of data across partitions.
Round-robin. The Integration Service distributes data evenly among all partitions. Use round-robin partitioning
where you want each partition to process approximately the same number of rows.
Dynamic Partitioning
If the volume of data grows or you add more CPUs, you might need to adjust partitioning so the session run time
does not increase. When you use dynamic partitioning, you can configure the partition information so the
Integration Service determines the number of partitions to create at run time.
The Integration Service scales the number of session partitions at run time based on factors such as source
database partitions or the number of nodes in a grid.
If any transformation in a stage does not support par titioning, or if the partition configuration does not support
dynamic partitioning, the Integration Service does not scale partitions in the pipeline. The data passes through onepartition.
4 Chapter 1: Understanding Pipeline Partitioning
-
7/27/2019 PC 901 AdvWorkflowGuide En
22/307
Complete the following tasks to scale session partitions with dynamic partitioning:
Set the partitioning. The Integration Service increases the number of partitions based on the partitioning
method you choose.
Set session attributes for dynamic partitions. You can set session attributes that identify source and target
file names and directories. The session uses the session attributes to create the partition-level attributes for
each partition it creates at run time.
Configure partition types. You can edit partition points and partition types using the Partitions view on the
Mapping tab of session properties.
Note: Do not configure dynamic partitioning for a session that contains manual partitions. If you set dynamic
partitioning to a value other than disabled and you manually partition the session, the session is invalid.
Configuring Dynamic Partitioning
Configure dynamic partitioning on the Config Object tab of session properties. Configure dynamic partitioning
using one of the following methods:
Disabled. Do not use dynamic partitioning. Defines the number of partitions on the Mapping tab.
Based on number of partitions. Sets the partitions to a number that you define in the Number of Partitionsattribute. Use the $DynamicPartitionCount session parameter, or enter a number greater than 1.
Based on number of nodes in grid. Sets the partitions to the number of nodes in the grid running the session.
If you configure this option for sessions that do not run on a grid, the session runs in one partition and logs a
message in the session log.
Based on source partitioning. Determines the number of partitions using database partition information. The
number of partitions is the maximum of the number of partitions at the source. For Oracle sources that use
composite partitioning, the number of partitions is the maximum of the number of subpartitions at the source.
Based on number of CPUs. Sets the number of partitions equal to the number of CPUs on the node that
prepares the session. If the session is configured to run on a grid, dynamic partitioning sets the number of
partitions equal to the number of CPUs on the node that prepares the session multiplied by the number of
nodes in the grid.
RELATED TOPICS:
Database Partitioning Partition Type on page 38
Rules and Guidelines for Dynamic Partitioning
Use the following rules and guidelines with dynamic partitioning:
Dynamic partitioning uses the same connection for each partition.
You cannot use dynamic partitioning with XML sources and target