Db2 for i Update - magic-ug.org · Major enhancements in Db2 for i deliver significant client...

© 2018 IBM Corporation© 2018 IBM Corporation

Db2 for i

Update

Doug Mack

Db2 for i Analytics Consultant

IBM Systems Lab Services

[email protected]

© 2018 IBM Corporation

Db2 for i •Standard compliant•Secure•Scalable•Functionally Advanced•Excellent Performance •Easier to use•Easier to maintain

IBM i 7.1

▪XML Support

▪Encryption enhancements (FIELDPROCs)

▪Result set support in embedded SQL

▪CURRENTLY COMMITTED

▪MERGE

▪Global variables

▪Array support in procedures

▪Three-part names and aliases

▪SQE Logical file support

▪EVI enhancements

▪Inline functions

▪CREATE OR REPLACE

▪DECLARE GLOBAL TEMPORARY TABLE

▪Catalog views

IBM i 7.2

▪Row and Column Access Control

▪XMLTABLE

▪CONNECT BY

▪TRANSFER OWNERSHIP

▪Named arguments and defaults for parameters

▪Obfuscation of SQL routines

▪Array support in UDFs

▪Timestamp precision

▪Multiple-action Triggers

▪Built-in Global Variables

▪1.7 Terabyte Indexes

▪Navigator Graphing and Charting

▪System Limits for IFS

IBM i 7.3

▪Temporal

▪Generated columns for auditing

▪OLAP Extensions

▪Regression Functions / Covariance

▪EVI Only Access

▪More Built-in Global Variables

▪More SQL Scalar functions

▪More IBM i Services

▪CREATE OR REPLACE TABLE

▪ATTACH & DETACH PARTITION

▪DBE tooling and automation

▪High priority customer requests

Value Proposition

How is IBM investing in Db2 for i

Increase the life expectancy and extend the value of legacy applications


Leverage available skillsReact with agility to changesPosition for the future

Faster response to requestsBetter data governanceImproved data access

Improved data analysis Empowered end usersBuild “system of insight”

Modularize for flexibilityImprove User InterfaceEnable for any HLL usage

Un-couple data access from RPGShift to SQL foundationEnforce business rules in databasePhase in - for minimal disruption

Modernize Query/400Leverage integrated BI solutionDeliver interactive reports, dashboardsVisualize data for better analysis

Reporting/Analytics

Modernization

IBM i Modernization Focus Areas

• Independently approached, interconnected topics

– Pursuing any one of these provides long term ROI

• Strategic and Tactical elements in each focus area

– Develop long term plan while delivering value to the business in the short term

Application

Modernization

Database

Modernization

Benefits

Approaches


Major enhancements in Db2 for i deliver significant client value:

o Temporal Tables – History of rows

Data-centric, easily deployed, robust SQL point-in-time capability

o Online Analytical Processing (OLAP) built-in functions

Adding more analytics capabilities directly into Db2 for i

o JSON Support

Easier integration w/ Db2 and applications

o IBM i Services – SQL for the Systems Administrator

Db2 for i 7.3


But first a quick look back…..


Row and Column Level Control (RCAC) – 7.2

• Not everyone is treated equal

– Patients

• Can only access their own data

– Physicians

• Can only access their own patients’ data

– Membership officers, Accounting, and

Drug researchers

• Can access all data

– Nobody else sees any data

1

2

3


Row01 REDRow02 BLUERow03 BLUERow04 REDRow05 GREENRow06 REDRow07 BLUERow08 BLUERow09 GREENRow10 GREENRow11 GREENRow12 GREENRow13 WHITERow14 GREENRow15 WHITERow16 GREENRow17 GREENRow18 GREEN

DB2RCAC

Actual Set

WHERE (CURRENT_USER = ‘USER1’AND COLOR = ‘BLUE’)

OR (CURRENT_USER = ‘USER2’AND COLOR = ‘RED’)

OR (CURRENT_USER = ‘USER3’AND COLOR = ‘WHITE’)

Row Permission


Row01 REDRow02 BLUERow03 BLUERow04 REDRow05 GREENRow06 REDRow07 BLUERow08 BLUERow09 GREENRow10 GREENRow11 GREENRow12 GREENRow13 WHITERow14 GREENRow15 WHITERow16 GREENRow17 GREENRow18 GREEN

DB2RCAC

Actual Set

CASEWHEN CURRENT_USER = ‘USER1’

‘********’ELSE

COLOREND

Column Mask


User1Row01 REDRow02 BLUERow03 BLUERow04 REDRow05 GREENRow06 REDRow07 BLUERow08 BLUERow09 GREENRow10 GREENRow11 GREENRow12 GREENRow13 WHITERow14 GREENRow15 WHITERow16 GREENRow17 GREENRow18 GREEN

DB2RCAC

Result Set Actual Set

SELECT ROW_NAME,CASEWHEN CURRENT_USER = ‘USER1’

‘********’ELSE

COLOREND

FROM A_SETWHERE (CURRENT_USER = ‘USER1’

AND COLOR = ‘BLUE’)OR (CURRENT_USER = ‘USER2’

AND COLOR = ‘RED’)OR (CURRENT_USER = ‘USER3’

AND COLOR = ‘WHITE’);

Row02 ********Row03 ********Row07 ********Row08 ********

User3 Row13 WHITERow15 WHITE

Mask

Permission


CURRENT_USER special register + VERIFY_GROUP_FOR_USER

SELECT CURRENT_USER FROM SYSIBM.SYSDUMMY1;

CREATE MASK SSN_MASK ON EMPLOYEE

FOR COLUMN SSN RETURN

CASE

WHEN (VERIFY_GROUP_FOR_USER(CURRENT_USER,'PAYROLL') = 1)

THEN SSN

WHEN (VERIFY_GROUP_FOR_USER(CURRENT_USER,'MGR') = 1)

THEN 'XXX-XX-' CONCAT SUBSTR(SSN,8,4)

ELSE NULL

END

ENABLE;

The CURRENT USER special register specifies the primary authorization ID

that is being used for statement authorization.

VERIFY_GROUP_FOR_USER function returns true (1) or false (0) by testing if

the user specified is in the group (i.e. a member of a group profile)


IBM Advanced Data Security for i (OS Option 47)

• Use iNav or SQL to implement (Alter Table, Activate)

• Must “secure” any triggers or functions

• Option 47 must be installed to:

– CREATE PERMISSION and CREATE MASK (RCAC)

– Open a table/file that has RCAC activated

Integration…• RCAC is applied after checking object authorization requirements

– Once passed the object authorization check:

• Row permissions can limit the set of rows accessed

• Column masks limit viewing of column data

• RCAC definition exist within the table object

• When ENABLED & ACTIVATED, RCAC is automatically applied by DB2

• RCAC is comprehensive and applies to any interface

(Native RLA, SQL, APIs, Commands, etc.)

• Only users with QIBM_DB_SECADM authority can manage RCAC

– Separation of roles and responsibilities


Major enhancements in Db2 for i deliver significant client value:

o Temporal Tables – History of rows

Data-centric, easily deployed, robust SQL point-in-time capability

o Online Analytical Processing (OLAP) built-in functions

Adding more analytics capabilities directly into Db2 for i

o JSON Support

Easier integration w/ Db2 and applications

o IBM i Services – SQL for the Systems Administrator

Db2 for i 7.3


With Temporal Tables, you can answer time-based questions:

o Who was the client rep as of two years ago?

o Who were the client reps over the last five years?

o Produce an inventory report using a different point in time

Benefit: Push the work to Db2, Remove management of History Data from HLLs

Db2 for i and Temporal Tables (7.3)

Current HistorySystem

Managed


Db2 for i – The Old Way: Do It Yourself

Current

Accessing Data• SELECT

Modifying Data• INSERT• UPDATE• DELETE

History

Accessing Data• SELECT

Modifying Data• INSERT• UPDATE• DELETE

UNION

Triggers / Programs


ALTER TABLE accountADD COLUMN row_birth

TIMESTAMP(12) NOT NULL IMPLICITLY HIDDENGENERATED ALWAYS AS ROW BEGIN

ADD COLUMN row_deathTIMESTAMP(12) NOT NULL IMPLICITLY HIDDENGENERATED ALWAYS AS ROW END

ADD COLUMN transaction_timeTIMESTAMP(12) IMPLICITLY HIDDENGENERATED ALWAYS AS TRANSACTION START ID

ADD PERIOD SYSTEM_TIME (row_birth, row_death)

CREATE TABLE account_hist LIKE account

ALTER TABLE account ADD VERSIONING USE HISTORY TABLE account_hist

Create history table

Establish birth/death of a row

Enable Temporal tracking

Temporal construction for data-centric history


o SQL statements just reference the current table

o New clauses on the SELECT statement

o FOR SYSTEM TIME AS OF <value>

o FOR SYSTEM TIME FROM <value> TO <value>

o FOR SYSTEM TIME BETWEEN <value> AND <value>

o New special register

o CURRENT TEMPORAL SYSTEM_TIME

Accessing a Temporal Table


With Db2 Temporal Tables, you can:

o Who was the client rep as of two years ago?

SELECT CLIENT_REP FROM ACCOUNTS

FOR SYSTEM_TIME AS OF CURRENT TIMESTAMP – 2 YEARS

o Who were the client reps over the last five years?

SELECT CLIENT_REP FROM ACCOUNTS

FOR SYSTEM_TIME FROM CURRENT TIMESTAMP – 5 YEARS

TO CURRENT TIMESTAMP

o Produce an inventory report using a different point in time

SET CURRENT TEMPORAL SYSTEM_TIME '2016-03-22 17:00:00';

CALL GENERATE_INVENTORY_REPORT();

Move Work to Db2 for i Example: Temporal Tables (7.3)


SQL OLAP Specification Extensions

On-Line Analytical Processing (OLAP) specifications provide the ability to return ranking, row

numbering, and other aggregate function information as a scalar value in a query result. RANK,

DENSE_RANK, and ROW_NUMBER were added in IBM i 6.1.

SELECT workdept, lastname, salary,RANK() OVER(PARTITION BY workdept ORDER BY salary DESC) rank,DENSE_RANK() OVER(PARTITION BY workdept ORDER BY salary DESC) dense_rank,ROW_NUMBER() OVER(ORDER BY workdept, lastname) row_number

FROM employee ORDER BY workdept, lastname

WORKDEPT LASTNAME SALARY RANK DENSE_RANK ROWNBR

AOO HAAS 52,750,00 1 1 1

AOO HEMMINGER 46,500.00 2 2 2

AOO LUCCHESSI 46,500.00 2 2 3

AOO O'CONNELL 29,250.00 4 3 4

AOO ORLANDO 29,250.00 4 3 5


SQL Query Engine and OLAP Specification Extensions

With IBM i 7.3, Db2 for i supports the following additional OLAP specifications for

summarizing and referencing rows within windows of result sets:

▪ Aggregates (SUM, COUNT, etc.)

▪ LAG and LEAD

▪ NTILE

▪ FIRST_VALUE, NTH_VALUE, and LAST_VALUE

▪ RATIO_TO REPORT

▪ CUME_DIST

What is the value of this?

▪ You could build analytical applications with much less complexity

▪ BI tools like Db2 Web Query and Cognos can “push the work” to Db2 for better performance


SQL OLAP Extensions Example

• Your Human Resources (HR) department wants to ensure salaries are equitable across the

company and across departments and there aren’t outliers or other discrepancies

– What is the employee’s salary compared to the average WITHIN their department?

– What is an employee’s ratio of salary within their department and overall company?

– For each employee, compare their salary to the two closest behind this person’s salary, and the

two closest ahead of them in salary


The Process

• Create a SQL Views (or script) using new 7.3 SQL Extensions

View for Salary compared to AVERAGEView for 2 ahead, 2 behind


Benefits

• Programmer Productivity

– A very complex data relationship challenge

becomes a SIMPLE SQL statement

• Performance

– Push the work to Db2 for i

• Leverage these functions with Db2 Web Query

for graphical representation or excel reports

– Simplified process for complex reporting challenges

• Get out of the business of creating extracts

and replicating data

– Reduce security risks, eliminate unneeded

processes


Db2 JSON Functions: Example with Watson Prototype

Db2 for i

Power Systems(IBM i)

Db2 Web Queryfor i

DataMigrator

Db2for i

Datamart

IBM Bluemix

Augment Data with Weather info

Visualize/deploy insights with Db2 Web Query

Order, Product, Inventory and measurements (revenue, COGS) Information

How many rain boots did we sell on sunny weekend days?


Rows and Columns (i.e. a “table”)

JSON “document”

JSON_TABLE function

Shredding the Returned JSON Data with the JSON_TABLE Function


Use JSON_TABLE function to Shred Data into Db2 Table(s)

Get Weather Almanac Information for Rochester Mn for Aug 10th

SELECT * FROM JSON_TABLE(SYSTOOLS.HTTPGETCLOB

('https://[email protected]/api/weather/v1/location/55901%3A4%3AUS/almanac/daily.json?units=e&start=0810' ,'') , '$'

COLUMNS(STATION_ID VARCHAR(100) PATH '$.almanac_summaries.station_id',STATION_NAME VARCHAR(100) PATH '$.almanac_summaries.station_name',ALMANAC_DATE VARCHAR(100) PATH '$.almanac_summaries.almanac_dt',INTERVAL VARCHAR(100) PATH '$.almanac_summaries.interval',AVERAGE_HIGH_TEMP VARCHAR(100) PATH '$.almanac_summaries.avg_hi' ,AVERAGE_LOW_TEMP VARCHAR(100) PATH '$.almanac_summaries.avg_lo',RECORD_HIGH_TEMP VARCHAR(100) PATH '$.almanac_summaries.record_hi',RECORD_HIGH_YEAR VARCHAR(100) PATH '$.almanac_summaries.record_hi_yr',RECORD_LOW_TEMP VARCHAR(100) PATH '$.almanac_summaries.record_lo',RECORD_LOW_YEAR VARCHAR(100) PATH '$.almanac_summaries.record_lo_yr',MEAN_TEMP VARCHAR(100) PATH '$.almanac_summaries.mean_temp',AVERAGE_PRECIPITATION VARCHAR(100) PATH '$.almanac_summaries.avg_precip',AVERAGE_SNOW VARCHAR(100) PATH '$.almanac_summaries.avg_snow‘ )

) AS ALMANAC_SUMMARIES;


Visualizations Provide Data Analysis including Weather Attributes

• Each day you get new data automatically

• Geographical Maps can be used to relate data to store location or city/postalcode

• Dashboards can answer various questions

– How many deliveries were LATE on rainy days?

26


PTF Services

QSYS2.GROUP_PTF_INFO – VIEW

QSYS2.PTF_INFO – VIEW

SYSTOOLS.GROUP_PTF_CURRENCY – VIEW

SYSTOOLS.GROUP_PTF_DETAILS – VIEW

http://ibm.biz/Db2foriServices

http://www.ibm.com/developerworks/ibmi/techupdates/db2/landscape

Security Services

QSYS2.AUTHORITY_COLLECTION – VIEW

QSYS2.AUTHORIZATION_LIST_INFO – VIEW

QSYS2.AUTHORIZATION_LIST_USER_INFO – VIEW

QSYS2.DRDA_AUTHENTICATION_ENTRY_INFO – VIEW

QSYS2.FUNCTION_INFO – VIEW

QSYS2.FUNCTION_USAGE – VIEW

QSYS2.GROUP_PROFILE_ENTRIES – VIEW

QSYS2.OBJECT_PRIVILEGES – VIEW

QSYS2.SQL_CHECK_AUTHORITY – UDF

QSYS2.USER_INFO – VIEW

SYSPROC.SET_COLUMN_ATTRIBUTE – PROCEDURE

Communication Services

QSYS2.NETSTAT_INFO – VIEW

QSYS2.NETSTAT_INTERFACE_INFO – VIEW

QSYS2.NETSTAT_JOB_INFO – VIEW

QSYS2.NETSTAT_ROUTE_INFO – VIEW

QSYS2.SERVER_SBS_ROUTING – VIEW

QSYS2.SET_SERVER_SBS_ROUTING – PROCEDURE

QSYS2.TCPIP_INFO – VIEW

SYSIBMADM.ENV_SYS_INFO – VIEW

Work Management Services

QSYS2.ACTIVE_JOB_INFO – UDTF

QSYS2.GET_JOB_INFO – UDTF

QSYS2.JOB_INFO – UDTF

QSYS2.JOB_QUEUE_INFO – VIEW

QSYS2.MEMORY_POOL – UDTF

QSYS2.MEMORY_POOL_INFO – VIEW

QSYS2.OBJECT_LOCK_INFO – VIEW

QSYS2.RECORD_LOCK_INFO – VIEW

QSYS2.SCHEDULED_JOB_INFO – VIEW

QSYS2.SYSTEM_STATUS – UDTF

QSYS2.SYSTEM_STATUS_INFO – VIEW

QSYS2.SYSTEM_VALUE_INFO – VIEW

System Health ServicesQSYS2.SYSLIMITS – VIEW

QSYS2.SYSLIMTBL – TABLE

Application Services

Journal Services

QSYS2.DISPLAY_JOURNAL – UDTF

QSYS2.JOURNAL_INFO – VIEW

Storage Services

QSYS2.ASP_INFO – VIEW

QSYS2.ASP_VARY_INFO – VIEW

QSYS2.MEDIA_LIBRARY_INFO – VIEW

QSYS2.SYSDISKSTAT – VIEW

QSYS2.SYSTMPSTG – VIEW

QSYS2.USER_STORAGE – VIEW

QSYS2.ENVIRONMENT_VARIABLE_INFO – VIEW

QSYS2.QCMDEXC – PROCEDURE

QSYS2.SERVICES_INFO – TABLE

QSYS2.SET_PASE_SHELL_INFO – PROCEDURE

QSYS2.STACK_INFO – UDTF

Librarian Services

QSYS2.LIBRARY_LIST_INFO – VIEW

QSYS2.OBJECT_STATISTICS – UDTF

Message Handling Services

QSYS2.HISTORY_LOG_INFO – UDTF

QSYS2.JOBLOG_INFO – UDTF

QSYS2.MESSAGE_QUEUE_INFO – VIEW

QSYS2.REPLY_LIST_INFO – VIEW

Java Services

QSYS2.JVM_INFO – VIEW

QSYS2.SET_JVM – PROCEDURE

IBM i Services

QSYS2.LICENSE_INFO – VIEW

SYSTOOLS.LICENSE_EXPIRATION_CHECK – PROCEDURE

QSYS2.OUTPUT_QUEUE_ENTRIES – VIEW

QSYS2.OUTPUT_QUEUE_ENTRIES – UDTF

QSYS2.OUTPUT_QUEUE_INFO – VIEW

Product Services

Spool Services

IBM i Services – SQL Interfaces to Common System Information

http://ibm.biz/Db2foriServices

http://www.ibm.com/developerworks/ibmi/techupdates/db2/landscape


Sample Reports in Db2 Web Query EZ-Install

• A dozen or so sample reports leveraging IBM i

Services are provided with EZ-Install

• Supports one or multiple LPARs/Servers

➢ System CPU Utilization

➢ Can be easily extended to other system metrics

• Auto Refresh Enabled

➢ Remote data collection requires remote system

connections to be set up (RDB directory entries

and SVRAUTE entries)

• Continues to collect and store historical data using

a Stored Procedure once the report is initially run

➢ Stops collecting if ‘no one is watching’ for an

extended period of time


Business Intelligence/Reporting Modernization with Db2 Web Query for i Version 2.2.1

• Integrated IBM i Business Intelligence and Data Warehousing

family of products

• Empower End Users with Data Driven Analysis

• Create Controlled Ad-Hoc Environment

• Build Data Integrity through Metadata and/or Data

Warehouse

• Integrate with Advanced Analytics in the IBM Cloud

• Get Started Quickly with Db2 Web Query EZ-Install

– Over 100 sample reports and supporting tutorials for the

business and I/T

– Query/400 Discovery Tool

– Systems Administrator sample monitoring reports Learn more at ibm.biz/db2webqueryi& db2webqueryi.blogspot.com

ibm.biz/db2webqueryi

db2webqueryi.blogspot.com

https://webfocusinfocenter.informationbuilders.com/wfbue/videos/Retail_Samples/Visualizations/sales_by_country_and_product.mp4


Integrating with Advanced Analytics: “Watson”

• The phenomenal growth in DATA – structured and unstructured – how can this be leveraged?

– The “Internet of Things” – a world full of sensors capturing MASSIVE amounts of data

– Unstructured data – e.g., social networks like Twitter, Blogs, Facebook, etc.

• New Inventions

– Face Recognition

– Machine or Deep Learning

– Personality Insights

– Natural Language Interpretation

– “Predictive” technologies

– Massively parallel systems and software stacks

– Industry Specific Customer Insight Solutions

• The advent of focused skills, job positions, and tools for them

– Business Analytics to Data Scientist to Chief Data Officers

• And put it all in the cloud to reduce barriers to entry and “pay as you go” utility pricing

IBM Bluemix


Db2 Web Query – Positioning for Advanced Analytics

Db2 for i

Power Systems(IBM i)Operational SystemShared Resources

Synonyms(metadata)

Operational Reporting &Dashboards

Db2 for i

Power Systems(IBM i)Data Warehouse

Dedicated Resources

System of Insight

Analytical Reporting &Dashboards

CleansedConsolidated

Secured!Automated, Repeatable

ETL

DataMigrator ETL

Neural NetworkMachine Learning

Db2 Cloud WarehouseCustomer Insights

Watson StudioAnd much, much

more

https://webfocusinfocenter.informationbuilders.com/wfbue/videos/Retail_Samples/Visualizations/sales_by_country_and_product.mp4

© 2018 IBM Corporation© 2018 IBM Corporation

Db2 for i and Db2 Web Query:

Foundations for Modernization

Doug Mack

Db2 for i Analytics Consultant

Db2 for i Update - magic-ug.org · Major enhancements in Db2 for i deliver significant client...

Documents

Transcript of Db2 for i Update - magic-ug.org · Major enhancements in Db2 for i deliver significant client...