Post on 25-Jul-2018
www.softwaretoolhouse.com
Complex BMC ITSM
Data Migrations
Software Tool House Inc. Ben Chernys Moderator/Presenter
Challenges of ITSM Data Migrations, ITSM Data, Migration Projects, Tools,
Using Meta-Update, Live Demo, Customer Case Studies
BMC RUG Dresden
Software Tool House
is a BMC Partner
2012-Oct-29
Rüdiger Tams Co-Moderator
NTT DATA Germany
www.softwaretoolhouse.com
Objectives
Agenda
Challenges of ITSM Data Migrations
An Introduction into ITSM Forms and Fields
Migration Projects: Typical Milestones and Challenges
Migration Tools & Methods Options
Using Meta-Update for Data Migrations
Live Demo: Real World Complexity CSV CMDB Import
Customer Case Studies
Services of Software Tool House and NTT DATA
www.softwaretoolhouse.com
Challenges of ITSM Data Migrations
Complex
Internal Structures Not Documented
Hierarchical trees of records in many
forms
Hidden Redundant References
Migrations
Migrating data across ITSM releases are significant
undertakings
Migrating legacy data or data from other systems is
almost impossible
Onboarding
Can be a complex, time consuming, expensive ARS or ITSM development task using staging forms and merge workflow and a host of tools.
Validations
Validating, Archiving, Merging,
Changing, Reporting, Extracting
hierarchical trees of data
– as a unit – is difficult.
ITSM Data Pain Points
www.softwaretoolhouse.com
Clarify to ITSM Challenges
Complex
Internal Structures Not Documented
Hierarchical trees of records in many tables
Hidden Redundant References
Select subsets of companies
Transforms
Login Ids Company Names
Site Names Relationship
Keys Status Fields Clarify Notes Task Types
Support Org, Groups, Staff
Transforms are often complex:
two or more Look Ups
Hierarchical
Clarify data, like ITSM data is a set of (more normalised)
hierarchical data –with different
keys and tables– which must be translated into
ITSM trees.
Ticket State
States must be set appropriately which means all requirements of ITSM workflow
must be met including
generating extra records in the
tree.
Often this means two updates.
ITSM Data Pain Points
www.softwaretoolhouse.com
Complex, Changing, Undocumented
7.6.3
2,314 Forms
194,790 Fields
7.6.04
2,794 Forms
235,196 Fields
8.0.0
3,188 Forms
249,109 Fields
ITSM Forms and Fields
Restrained, linear progression: New functionality, enhancements, maintenance.
Sign of maturity.
www.softwaretoolhouse.com
Fixed Price ARS Data Migrations
Tests
Scripts Analysis
Data Migration Projects
Tasks
Milestones
Typical
www.softwaretoolhouse.com
Data Migration Projects Tasks Milestones Description
Data
Analysis
Running of scripts and SQL queries to collect reports on productions data. Q & A on data
relationships, migration needs.
Data
Summary
Inventory of production data. Spreadsheets and documents listing forms, records counts, sets of
different data statistics (CMDB by Asset Type, PCT, etc.)
Data Work
Shops
Q & A on data relationships, migration needs with data users and managers.
Migration
Summary
Documents the data that to be migrated and to be left. Built from the Data Summary.
Transform
Work
Shops
Required data transforms are identified, methods described, and any mapping LookUp spreadsheets
identified.
Mapping
Sheets
Documents transforms, methods, and LookUp sheets. Sheets are populated with source / Traget data
and instructions.
Mapping
Sheets
Developed
Customer responsibility with guidance. Continuously tested as scripts developed.
Script
Developed
Scripts are modified and developed to implement all Mapping transforms. Small subsets of data are
often migrated and updated.
Scripts
Delivered
Scripts are ready and a Usage Guide is published listing the script running order and dependencies.
Mapping
Sheets
Completed
Customer completes all transform sheets.
Migration Tests are scheduled.
Full
Migration
Tests
Full data is migrated according to the Usage Guide to measure correctness, completeness, volume,
performance.
Migration
Plan
A step by step plan is documented for the production migration weekend or longer if required because
of the volume and performance parameters.
Migration
Weekend
The Migration plan is carried out step by step. When complete, customer people validate the data and
make a Go / No Go decision.
Post
Migration
Support
Immediate support for any data issues on new system. Missing records investigated, LookUps or
Source corrected, and migrated.
Migration
Results
Summary of data migrated, immediate post production support. Guide to corrective LookUp changes
for remaining missing records.
Migration
Support
Assistance with individual runs and LookUp corrections.
www.softwaretoolhouse.com
ITSM Forms and Fields
ARS Import Export
Manual exports to
CSVs, Excel to change values, manual imports.
Automated with staging forms and
import workflow: a
time-consuming, expensive Workflow Project.
Web Services
Real time, retries,
queuing, logging is
implementation (staging
form) dependent
Automated with staging forms and
import workflow
similar to ARS Import / Export
Migrator DSO
Automated with complex
Merge workflow.
May require changes to application.
Workflow Project can
approach API costs.
AIE AI
Limited mapping
capabilities.
Fixed targets: multiple
Exchanges / Outputs.
Point & Click interface
cumbersome to set
qualifications, conditions.
Needs Recon jobs.
Hard to debug.
API Perl
Java c
Sure way to automate.
Expensive and slow to
develop.
Usually needs outside, non-Remedy staff.
SQL
Dangerous to the integrity of
the ARS system data.
Expensive and slow to
develop.
Usually needs outside, non-Remedy staff.
www.softwaretoolhouse.com
Meta-Update In a Nutshell
ETL – Extract Transform Load – Scripting tool
• Meta-Update allows any BMC Remedy administrator / developer to automate any ETL data operation, using the API, with no need for API or
programming skills at all!
with no staging forms,
workflow, customizations, or
server changes at all!
• With Meta-Update, one develops proven, repeatable scripts that can be used over and over again,
www.softwaretoolhouse.com
Easy, Quick, Robust — Savings for any Remedy shop
Meta-Update In a Nutshell
Simple
One single “ini” file
No Staging forms and
Merge workflow
Scripts can be done in minutes
Time To Market
Order of magnitude
reductions in time and efforts
Clean
Uses BMC Remedy API
No ARS / ITSM server
changes
No -ITSM- Customization
Auditable
Logging with full control
Configurable Record Audits
Remedy Client Logging
www.softwaretoolhouse.com
Meta-Update In a Nutshell
Meta-Update — extends the concept of a field reference and adds value transform features.
• ARS: $ Field $ • Meta-Update: $ Tag, Field $
With a few simple words, queries and files are iterated, records looked up, references are made and used, and with all these Tags in memory at the same time, updates are made and files are output.
www.softwaretoolhouse.com
Meta-Update Jobs
How Meta-Update Helps
Migrations
ITSM releases, Legacy apps
Imports
On-boarding, foundation data, CMDB data
Corrections
Change, merge, adjust data at will
Extracts
Save and create files of any kind
Integrations .
Validations
www.softwaretoolhouse.com
Meta-Update script functions gives you better control over your data manipulation tasks.
Meta-Update is a script based solution. It is easier to learn than a "pure" development language like Java or C#. People who are used to this should be able to do scripting after some days.
Meta-Update "hides" necessary function calls you normally need when using a development language.
You can combine the scripts with e.g. ARS escalations to run them on a time triggered level.
Re-use of scripts possible and with this a fast adaptability to new requirements.
With a Meta-Update license different additional helper-tools are delivered to make your work easier.
Experiences and Impressions with Meta-Update
in a multi-national company project
www.softwaretoolhouse.com
Some points to keep in mind:
If not structured accordingly you could loose overview in big scripts – so stick to the recommended structure.
When developing different scripts in a team, you should use a Source Control System like SVN or MS Visual SourceSafe
Performance of data transfer: C API gives you a limit – but Meta-Update already caches data structures to make the best out of it.
“Meta-Update can give you much more power and control over your BMC Remedy ITSM installation and in combination with existing tools, like BMC Remedy AI, it can make your life much easier, e.g.
when you need to handle many data manipulations.”
Rüdiger Tams, NTT DATA Germany
Experiences and Impressions with Meta-Update
in a multi-national company project
www.softwaretoolhouse.com
A UsedBy and OwnedBy column contain lists
and need to generate appropriate associations.
Needs 2nd CSV to
get Class, CTI.
Lists of Login IDs
need to build CI
Associations.
Class dependent
attributes.
BMC_Disk “Size”
attribute is a decimal.
Needs normalization.
Live Demo
Time and effort savings
Sample CSV CI Import
www.softwaretoolhouse.com
Case Study: CSV CI Import
Needs 2nd CSV to
get Class, CTI.
Time and effort savings
Sample CSV CI Import
www.softwaretoolhouse.com
Live Demo
Live Demo
www.softwaretoolhouse.com
Reference: Logica UK
Case Studies
Meta-Update migration scripts were entirely customer written by their off-shore team directed by the UK Transition Manager after attending a one-hour WebEx meeting with two colleagues.
“Heavily customised” ITSM 5 → ITSM 7 data migration comprising Foundation and Transaction (Help Desk, Changes, Assets) for 400+ customers including 1500+ SLAs.
"We had a mammoth task of migrating 400+ companies from ITSM 5.5 to ITSM 7.1. Initially this task looked very complicated as the 5.5 system was highly customized to suit individual customer's requirements and the migration into mismatched schemas looked impossible. 1500 SLAs for these customers added to the complications.
“But when we started evaluating Meta-Update, we saw some light. We can now migrate SLAs, implement complicated transformation rules, stamp mapping values by looking up external configuration files, read from multiple servers, etc. Best part is that once Meta-Update scripts are done and tested, you can repeat it for any number of times.
“I recommend Meta-Update for all those who want the flexibility of a scripting language to handle bulk ARS data."
Transition manager:
"We used Meta Update to migrate Foundation, Asset, SLM and Ticket data from ITSM 5.5 to ITSM 7. Meta Update enabled us to perform this complex migration for many customers in record time. Without Meta Update. we would not have been able to achieve this task.
Project manager:
www.softwaretoolhouse.com
Reference: Dataport
Case Studies
• 35K Changes
• 65K CIs
• 120K Help Desk Tickets
• 120K Associations
• 80K Custom Records
ITSM 6 → 7 “Transaction Data” Migration
• Milestone based payments
Fixed price Monday morning, all open tickets and changes were ready for work with attachments and diary fields in work logs, all associations in place, assigned to the correct people and groups.
Migration across a weekend. Significant foundation data changes: Support Groups, Sites, Organisation data; substantial data cleansing; some missing requesters and support staff – all automated.
www.softwaretoolhouse.com
Reference: Met Office
Case Studies
Problem that had been in place for one year solved in a week with Meta-Update.
A single Meta-Update script read a control CSV that gave source CTI and new target classes. Script fixed Product Catalogue, replicated CIs, Work Logs, audit records, associations and relationships. Then ran with a –del to delete original CIs.
“Bad ITSM migration” caused 12,000 non-computer systems in CMDB Class Computer System. Worked a year: associated with tickets, changes, contracts, people, related to other CIs.
www.softwaretoolhouse.com
Reference: Multi-national in Germany
Case Studies
Developed in 1 man * 2 months. Clarify shut down on Friday. All tickets being worked on in ITSM on Monday. All historical data in place.
Migration of a running Clarify Help Desk into a production ITSM 7.6 Incident, Problem, Task.
Meta-Update loaded all foundation data including People, Support Staff, Company Organisation, Sites, Operational Catalogue, Product Catalogue, CMDB CIs & relationships.
New ITSM 7.5 full suite implementation. Migration from Customer Developed CMDB & Identity Management Database application with over 200 tables, as well as Support Staff, rights, groups, structure, Change & Task Templates, Approval Groups and members from 4 independent ITSM 6 servers.
Clarify Help Desk ITSM 6 & Bespoke CMDB & Identity Mgmt.
www.softwaretoolhouse.com
More Information
This Presentation
Meta-Update
Management Summary
More Details
This presentation is available as a link on our home page.
Meta-Update Management Summary Windows Movie (12 mins.)
Links
These links take you to our web site.
www.softwaretoolhouse.com
Next Steps www.SoftwareToolHouse.com
Check out the web site
Pick up a trial license with our Live Install and
Scripting 101 presentation
Download the Scripting 101 presentation and
the Meta-Update User’s Guide
Look at the samples in the Script Library
Contact Us at www.softwaretoolhouse.com
Ben Chernys
+49 171 380 2329
Rüdiger Tams
+49 151 1680 2589