Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)
BIML: BI to the next level
-
Upload
davide-mauri -
Category
Technology
-
view
706 -
download
1
Transcript of BIML: BI to the next level
![Page 1: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/1.jpg)
13.12.2014
BI Markup Language:
BI to the next level
Davide Mauri
SolidQ – Founder & Mentor
@mauridb
www.solidq.com
![Page 2: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/2.jpg)
13.12.2014
Thank you to all our SPONSORS!
![Page 3: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/3.jpg)
13.12.2014
Davide Mauri
20 Years of experience on the SQL Server Platform Specialized in Data Solution Architecture, Database Design,
Performance Tuning, Business Intelligence
Projects, Consulting, Mentoring & Training Regular Speaker @ SQL Server events
Microsoft SQL Server MVP
President of UGISS (Italian SQL Server UG)
Mentor @ SolidQ
Find me here: Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
Twitter: @mauridb
![Page 4: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/4.jpg)
13.12.2014
Agenda
BI, DWH & ETL today
BIML – What’s that?
Tools
BIDS Helper
MIST
BIML Kick Off
Basic Concepts
Automation With “Embedded” C#
BIML & PowerShell
![Page 5: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/5.jpg)
13.12.2014
Building a BI/DWH/ETL in 2014
Is still a (almost) manual process
A *lot* of repetitive low-value work
No (or very few) standard tools available
![Page 6: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/6.jpg)
13.12.2014
How it should be
Semi-automatic process
“develop by intent”
Define the mapping logic from a semantic
perspective
Source to Dimensions / Measures
(Metadata anyone?)
Design the model and let the tool build it for you
![Page 7: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/7.jpg)
13.12.2014
How it should be
Semi-automatic process
“develop by intent”
Define the mapping logic from a semantic
perspective
Source to Dimensions / Measures
(Metadata anyone?)
Design the model and let the tool build it for you
CREATE DIMENSION Customer
FROM SourceCustomerTable
MAP USING CustomerMetadata
ALTER DIMENSION Customers
ADD ATTRIBUTE LoyaltyLevel
AS TYPE 1
CREATE FACT Orders
FROM SourceOrdersTable
MAP USING OrdersMetadata
ALTER FACT Orders
ADD DIMENSION Customer
![Page 8: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/8.jpg)
13.12.2014
The perfect BI process & architecture
Iterative!
![Page 9: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/9.jpg)
13.12.2014
Design Pattern
“A general reusable
solution to a
commonly occurring
problem within a
given context”
![Page 10: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/10.jpg)
13.12.2014
No Monkey Work!
Let the people think
and let the machines
do the «monkey»
work.
![Page 11: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/11.jpg)
13.12.2014
BIML – What’s that?
Business Intelligence Markup Language
XML-Based
BI Object-Generator
SQL Server
Integration Services
Analysis Services
![Page 12: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/12.jpg)
13.12.2014
BIML – What’s that?
Allows a declarative approach to define BI
objects
Favor automation
Human-readable
Compatible with SCC system
Allows merge of files
Born from “Project Vulcan”
Still available on CodePlex
![Page 13: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/13.jpg)
13.12.2014
BIML – What’s that?
Once objects are generated, BIML is not needed
anymore
No additional runtime/components required
Support for SQL Server 2008 & 2012 objects
2014 is coming very soon
Anyway generated packages are automatically
converted into 2014 packages by Visual Studio 2013,
without problems
![Page 14: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/14.jpg)
13.12.2014
BIML – What’s that?
BIMLScript is based on the T4 templating engine.
Allows the generation of BIML using
C#
VB.NET
BIML + BIMLScript = HTML + ASP Classic
![Page 15: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/15.jpg)
13.12.2014
Tools
BIDS Helper
http://bidshelper.codeplex.com/
Visual Studio Plugin
Free
Source Code Available
Not all BIML features are supported
“Limited” to SSIS generation only
![Page 16: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/16.jpg)
13.12.2014
Tools
MIST
Full-Featured BIML IDE
Buy or Rent
Made by BIML creator: Varigence
![Page 17: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/17.jpg)
13.12.2014
DEMO
First Contact with BIML
![Page 18: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/18.jpg)
13.12.2014
Basic Concepts
A single BIML file can generate more than one
SSIS Package
All SSIS object and related properties are exposed
via BIML elements
Additional specific attribute: ConstraintMode
Linear or Parallel
![Page 19: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/19.jpg)
13.12.2014
Basic Concepts
Parallel Constraint Mode
“Inputs” connected to “Outputs”
SSIS Variables: “dot” notation
![Page 20: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/20.jpg)
13.12.2014
Basic Concepts
Control Flow and Data Flow 100% supported
In Data Flow it’s mandatory to define input and
outputs
Since they aren’t constraints, but data streams
![Page 21: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/21.jpg)
13.12.2014
DEMO
BIML is your friend!
![Page 22: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/22.jpg)
13.12.2014
Automation with BIMLScript
BIML Script (T4) Directives: <#@ and #>
![Page 23: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/23.jpg)
13.12.2014
Automation with BIMLScript
BIML Script Tags: <# e #>
Allows usage of C# / VB.Net within BIML
The “=” symbol allows “in-line” code injection
![Page 24: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/24.jpg)
13.12.2014
Automation with BIMLScript
Metadata is needed in order to define what to
generate
Metadata can be extracted by
BIML Engine (Hadron)
T-SQL Queries
![Page 25: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/25.jpg)
13.12.2014
Automation with BIMLScript
Metadata via BIML Engine (Hadron)
![Page 26: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/26.jpg)
13.12.2014
Automation with BIMLScript
Metadata via SQL Query
![Page 27: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/27.jpg)
13.12.2014
DEMO
BIMLScript RULES!
![Page 28: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/28.jpg)
13.12.2014
Debugging BIML
With complex BIML Script it can be difficult to
“see” what’s happening behind the scenes
With MIST you can see the resulting BIML
generated from BIML Script
But with BIDS Helper you’re on your own.
NLog can help a lot here!
![Page 29: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/29.jpg)
13.12.2014
Debugging BIML
Nlog: free, powerful & flexible logging infrastructure
http://nlog-project.org/
Stand-Alone Assembly
Just reference the assembly in the BIML Script (T4) template and then use Nlog objects & methods
http://sqlblog.com/blogs/davide_mauri/default.aspx
![Page 30: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/30.jpg)
13.12.2014
DEMO
BIMLScript & NLog
![Page 31: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/31.jpg)
13.12.2014
BIML & PowerShell
What if you don’t like C#...
Or if BIMLScript looks to complex?
Especially without MIST?
One option is to generate BIML using PowerShell
Is just XML after all….
![Page 32: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/32.jpg)
13.12.2014
DEMO
BIML & PowerShell
![Page 33: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/33.jpg)
13.12.2014
Conclusions
Use BIML!!!!
It helps to
Make better use of your time
Create your own ETL patterns
http://www.slideshare.net/davidemauri/automating-dwh-
patterns-through-metadata
Support Refactoring & Enables Agility
Favor use of metadata
![Page 34: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/34.jpg)
13.12.2014
References
BIML Script
http://bimlscript.com/
Stairway to BIML
http://www.sqlservercentral.com/stairway/100550/
Automating DWH Patterns Through Metadata
http://www.slideshare.net/davidemauri/automating-
dwh-patterns-through-metadata
![Page 35: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/35.jpg)
13.12.2014
Q&A
Questions?
![Page 36: BIML: BI to the next level](https://reader033.fdocuments.net/reader033/viewer/2022060205/55a1e04c1a28ab27778b4769/html5/thumbnails/36.jpg)
13.12.2014
Thanks!