Post on 12-Feb-2022
4/9/20101
Atrium Integration Engine: What Is It? How Do You Use It?
› Atrium Integration Engine (AIE) Overview– What is it?– Overview of latest features
› Tell me how to do it– Under the hood details of creating CMDB Integration using AIE
© Copyright 4/9/2010 BMC Software, Inc.2
BMC Atrium Integration Engine
› What is it?– A mechanism to manage data transfers between external data sources and
CMDB (also AR Forms)
– Bi-directional and multi-threaded data transfer engine
– An Enterprise Application Integration tool with data transformation capability
– It supports event-based or scheduled information flow
– Ideal for bulk, incremental, data synchronization transfers
– Provides interfaces to external data source and the CMDB
– Provides an intuitive graphical interface for field mapping configuration
© Copyright 4/9/2010 BMC Software, Inc.3
BMC Atrium Integration Engine
› How is it used?– Transform data to the CMDB data model– Efficiently load data in bulk– Can create relationships between instances as they are inserted in the CMDB.
› Advantages– Designed for complex integrations– Easily configurable– Can be used against a variety of applications– Extensible
› When Is AIE a Fit?– Complex transfers requiring data transformation – Handling large amount of data transfers– Multiple integration points
© Copyright 4/9/2010 BMC Software, Inc.4
BMC Atrium Integration Engine 7.1
› New Name and Repackaging for EIE› Released in late August 2007› Broad Enhancement Areas
– Performance and Scalability• Improved performance of Relationship exchanges• Better memory management for all update operations
– Usability Improvements• Lot of usability improvements (please see the release notes for details on new features)
› Packaging– Ships with Atrium CMDB 2.1 to integrate with Atrium CMDB– One product, multiple adapters
• Oracle • SQL • DB2 New!• XML New!
– XML adapter only supports class based relationships (not vendor)• Flat file Improved!
© Copyright 4/9/2010 BMC Software, Inc.5
How to Create AIE Based Integrations to CMDB: The Ingredients To Success
› Data Structure Knowledge› External Data from Discovery/Inventory› Atrium Integration Engine› Atrium CMDB› Reconciliation Rules› Workflow Filters – if Needed
© Copyright 4/9/2010 BMC Software, Inc.6
AIE Terminology
› Data Exchange Application (DEA)– AR System based application at the heart of AIE
› AIE Service– DEA passes the service instructions on how to interact with the adaptor
› Data Mapping– CMDB Class/AR Form mapped to External Table– CMDB Attributes (or fields of AR Form) mapped to Columns of External Table– Values mapped to External Table Columns
› Exchange– Collection of “similar” Mappings– Definition of when to run– “Pipe that manages the flow”
› Adapter– The Plug at the end of the wire
DataExchangeApplication
Data Mappings
Data ExchangesAdapter
AIEService
© Copyright 4/9/2010 BMC Software, Inc.7
Discovery Environment Remedy Environment
Configuration SQL Example
SQL Database
DiscoveryAR System
(can be on either server)
DataExchangeApplication
Database for AR System
CMDBDiscovery
Dataset
Data Cleansing Workflow
CMDBBMC.ASSET
Dataset
RemedyAsset
Management
CMDBReconciliation
Engine
CMDBReconciliation
Rules
Data Mappings
Data ExchangesAdapter
AIEService
© Copyright 4/9/2010 BMC Software, Inc.8
External Data Source
› Database data› XML data› Flat file data› Source Relationships can be stored as
– foreign key– relationship tables
© Copyright 4/9/2010 BMC Software, Inc.9
Atrium CMDB
› Configuration Management Database– Store multiple datasets– Reconcile datasets– Store Relationships– Base data for OOTB Applications
› Data Sets– Data value in a field... but so much more– Data Sets you should use
• BMC.ASSET – The “Gold Standard”• BMC.SANDBOX – Temp location for GUI changes to Assets• VENDOR.PURPOSE.PRODUCT – custom datasets
– Examples: BMC.IMPORT.MARIMBA, MS.IMPORT.SMS, ACME.IMPORT.LEGACY
© Copyright 4/9/2010 BMC Software, Inc.11
Configuration > Database Field Menus
› Populates the “vendor” table and column menus› Run before you start any mapping work
1. Load
2. Add Desired Tables
3. Create Menus
© Copyright 4/9/2010 BMC Software, Inc.12
How To – Data Mapping
› Mapping Name = name for new mapping› Direction = External Data into CMDB› Namespace = BMC.CORE in most cases› Class = Destination for data, pick from the list› External Data Store = SqlServer, Flatfile or Oracle› Table Name = Source of data (pick from list)
– if not in the list, run rulehelper.exe to create the menu items.› Row Element = (XML only) the element that represents a row› Click “Create” to save
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.13
How To – Data Mapping
If the Table Name does not show in the drop down list, it is because you have not configured the Database Field Menus from the main console
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.14
How To – Data Mapping
› Used to map source and destination side primary/foreign keys› Keys must be unique on both sides
– If multiple keys, stored as AND to define Unique
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.15
How To – Data Mapping
› Map appropriate CDMB Class.Attribute to a Data Store Attribute› Make sure to specify CMDB mandatory fields such as
– Fields use in Key mappings– Data Set ID (automatic if “Use Dataset” is checked)– Short Description – Submitter – Name = anything that identifies the entry– AssetLifeCycleStatus = 3 (if Asset Management is installed)
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.16
How To – Data Mapping
› Rules – Works like “Function” key in Remedy User and Administrator Tools– Select by highlighting the row and click “Add”– Rule is added– Refer to the Rules box for hints on how to complete the rule
› Available Rules
© Copyright 4/9/2010 BMC Software, Inc.17
How To – Data Mapping
› Only works if you can write to the Vendor / Source Database› Can be any value› Usually is completion status
– 0 = transfer with no error– N = error # - corresponds to ARERROR
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.18
How To – Data Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete– Set MarkAsDeleted to 1 (Yes)– Other attributes may be set as desired
› If you choose to “Permanently Delete” – Hard Delete– No field to set. AIE will do trigger the deletion
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.19
How To – Data Mapping
- Use to limit amount of data in a given transfer. Important if you are experiencing performance or memory issues
- If you specified “Restrict to Dataset” on the Main Tab, you don’t need to include it here.
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.20
How To – Data Mapping
1. If set to all data that can be transferred will be. This could result in partial data records.2. Transfer as NULL cleans up any data that might be mistaken for null in a field3. If trailing blanks are possible in your key values, choose to consider4. Stores change history in a diary field you designate. Attribute must exist in CMDB.5. If Yes, updates are only made when a change is detected in the source data
› You must select a field that can store a negative integer to store the checksum6. Allows you to retrieve new or change data since last transfer
› You must select a date/time field to store last transfer timeData Exchange
Data Mapping Relationship Exchange
Relationship Mapping
123456
© Copyright 4/9/2010 BMC Software, Inc.21
How To - Data Exchange
› Exchange Name – Name you create to identify the exchange› Instance Name – Specify the AIE thread to use› Data Exchange Type – Specify the type of mapping contained in the exchange› External Data Store – Must match the mapping(s)› Execute Exchange As
– Schedule Only• Use Schedule tab
– Event Driven Only• run eiexfer.exe from command line, filter or active link
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.22
How To - Data Exchange
› Select the mapping(s) to include in the Exchange container.› Multiple Mappings
– Cuts down on queries for source or target› Main Mapping
– Need to designate if multiple mappings included– Key values only come from Main Mapping
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.23
How To - Data Exchange
› Interval– Works like the interval function in escalation workflow
› On Completion Section– When this exchange is done, you can configure
another exchange to run
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.24
How To - Data Exchange
› Time– Works like the time function in escalation workflow– Set day and time to run
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.25
How To - Data Exchange
› To Be Triggered– Use this option when you want to exchange to run on
completion of some other exchange– Must list the exchange in the On Completion section
on the other exchange
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.26
How To - Data Exchange
You can either set the external connection information on each exchange or on the Connections link from the Data Exchange Console
Alternate Method
Use if the AR Server involved in the exchange is NOT the server that DEA is installed on set to “No” and fill in the info.
Advantage – 1 license/DEA can service Dev, Test & Production
© Copyright 4/9/2010 BMC Software, Inc.27
How To - Data Exchange
› While Testing– Limit records so you have less to undo
› In Production – Set Enable Debug to No
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.28
How To – Relationship Mapping
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
Data Mapping
Data Mapping
Vendor Relationship Mapping
•Map using External Data Store Mapping.•Trigger on Complete of Data Exchange.
V_Computer
V_DiskDrive
Vendor Relationship Table
BMC_ComputerSystem
BMC_DiskDrive
BMC_HostedSystemComponent
© Copyright 4/9/2010 BMC Software, Inc.29
How To – Relationship Mapping
Data Mapping
Class Relationship Mapping
•Map using CI Class Relationship Mapping•Put in an Event Driven Only Exchange*
*In this case, the relationship dataexchange only runs after the data transfer for the two configuration item classes has completed.
V_Computer
V_DiskDrive
BMC_ComputerSystem
BMC_DiskDrive
BMC_HostedSystemComponent
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
Data Mapping
© Copyright 4/9/2010 BMC Software, Inc.30
How To - Relationship Mapping
› This is where you define which 2 classes to relate and which relationship› Must Select Relationship Source on Create› External Data into CMDB is the only valid Direction› If you choose Relationship Source of “External Data Store”
– must specify DB2, FlatFile, Oracle, SqlServer– Table Name
› It is best to Restrict to DatasetID and specify
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.31
How To - Relationship Mapping
› Define mapping criteria that defines the relationship› Works just like creating Relationship in CMDB
– Source on Left, Target on RightData Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.32
How To - Relationship Mapping
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
› Used to define values for a relationship instance› Mandatory attributes shown above must be mapped› To get a list of Attributes of each CI Class choose class from drop down.
Attributes list will then appear
© Copyright 4/9/2010 BMC Software, Inc.33
How To - Relationship Mapping
› Used to define values for a relationship instance› Mandatory attributes shown above must be mapped› To get a list of Attributes of each CI Class choose class from drop down.
Attributes list will then appear
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.34
How To - Relationship Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete– Set MarkAsDeleted to 1 (Yes)– Other attributes may be set as desired
› If you choose to “Permanently Delete” – Hard Delete– No field to set. AIE will do trigger the deletion
© Copyright 4/9/2010 BMC Software, Inc.35
How To - Relationship Mapping
› Use if you are doing class based mapping› Should match query used on class mapping› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.36
How To - Relationship Mapping
› Use if you are doing class based mapping› Should match query used on class mapping› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.37
How To - Relationship Mapping
› Use if you are doing class based mapping› Should match query used on class mapping› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.38
How To - Relationship Mapping
› Used only if a External Data Store mapping› This is used to limit the external records
returnedData Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.39
How To - Relationship Exchange
› Works the same as a normal Data Exchange› Data Exchange Type = Relationship Class
Data Exchange
Data Mapping Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.40
General AIE Hints
› When testing an exchange –– Test one exchange at a time (when done chain them to be triggered)– Use Run Now button to trigger an exchange immediately– Set Limit = 1 (so that you only run once)– Be sure you know your debug path
› When writing a Vendor Query –– Test it with Direct SQL – If it doesn’t work there, it won’t work here– Use Indexed fields if possible
This won’t work – MacAddress != NULLThis will – MacAddress IS NOT NULL
© Copyright 4/9/2010 BMC Software, Inc.41
General AIE Hints
› When setting an AR System field in a mapping –– Doesn’t work like a setfield or pushfield operation– Must Use Rules to convert functions, keywords, etc.
Example of ConcatenationThis won’t work – $Manufacturer0$+”-”+$MediaType0$This will – function|concat($Manufacturer0$,”-”,$MediaType0$)
Example of an Integer FieldThis won’t work – 0This will – Constant|0
© Copyright 4/9/2010 BMC Software, Inc.42
Atrium CMDB
› Reconciliation Rules– Identify
• Look for potential duplicates– Compare– Merge– Cleanup / Purge
• Can delete entire data set from all tables• Can delete selected items
– Configuration Item Set field MarkedForDelete = Yes– Create Reconciliation Rule to Purge if field is set
» Define how to handle related items– Run as a Reconciliation Job
CMDB 3rd PartyDataset
CMDB Reconciliation
Engine
CMDB Reconciliation
Rules
CMDB MasterDataset