Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools...

57
1 1998 Executive Information Systems, Inc. Executive Information Systems, Inc. Architectural Evolution in Data Warehousing: The Coming of Distributed Knowledge Management Architecture (DKMA) By Joseph M. Firestone, Ph.D. [email protected] September 9, 1998 EIS Web Site: http://www.dkms.com

Transcript of Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools...

Page 1: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

1

1998 Executive Information Systems, Inc.

Executive Information Systems, Inc.

Architectural Evolution in DataWarehousing: The Coming of Distributed

Knowledge Management Architecture (DKMA)

By

Joseph M. Firestone, [email protected]

September 9, 1998

EIS Web Site: http://www.dkms.com

Page 2: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

2

8The Dynamic Integration Problem is the problemof proactively and automatically monitoring andmanaging evolutionary change in datawarehousing systems without imposing atraditional and constraining “Top-Down”architecture.

8 It is the problem of providing managers of bothdata warehouses and data marts the power toinnovate, while still maintaining the integrationand consistency of the system.

Dynamic Integration and Data Warehousing

1998 Executive Information Systems, Inc.

Page 3: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

3

Dynamic Integration andArchitectural Evolution in Data Warehousing

8 Data Warehousing is now a complex systemsintegration problem. A full-blown DataWarehousing System may encompass8 the following database servers:

8 The data warehouse

8 various data marts (department, function, orapplication-specific DSSs, using Relational, Multi-dimensional (MOLAP), or Column-based Servers)

8 One or more Operational Data Stores (ODSs)

8 One or more Data Staging Areas

1998 Executive Information Systems, Inc.

Page 4: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

4

Dynamic Integration and ArchitecturalEvolution in Data Warehousing (Two)

8 the following application servers8 Web Servers

8 ETML Servers

8 Data Mining servers

8 Stateless Transaction Servers (e.g., MTS, Jaguar CTS,etc.

8 Business Process engines (e.g. Persistence Power-Tier,DAMAN InfoManager)

8 Document Servers

8 ROLAP Support Servers (e.g., MicroStrategy,Information Advantage)

8 Report Servers

8and various front-end OLAP and reporting tools

1998 Executive Information Systems, Inc.

Page 5: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

5

Dynamic Integration and ArchitecturalEvolution in Data Warehousing (Three)

8 The Dynamic Integration problem in the contextof this complexity of components and interactionsis three-fold:8First, we need an integrated view of all server-

based assets;

8Second, we need to manage flows of data,information, and knowledge throughout this systemto maintain the common view in the face of changein form and content, and to distribute the system’sdata, information, and knowledge bases asrequired, and

1998 Executive Information Systems, Inc.

Page 6: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

6

Dynamic Integration and ArchitecturalEvolution in Data Warehousing (Four)

8Third, we need such management to occurautomatically and without centralizing the systemso that the authority and responsibility for addingnew data and information to the system isdistributed.

8Automated dynamic integration is a capability notnow provided by data warehousing vendors. It isincreasing recognition of the need for thiscapability that drives architectural evolution in theDW system.

1998 Executive Information Systems, Inc.

Page 7: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

7

Architectures forManaging DSS Integration

8 Here are six architectures for managing andviewing the problem of integration.

8 Top-Down Architecture(Inmon, Prism Solutions, Carleton Corporation, ETI)

8 Bottom-Up Architecture(Broadbase Information Systems, Sagent, ArdentDataStage)

8 Enterprise Data Mart Architecture (EDMA)(Informatica, Carleton)

8 Data Stage/Data Mart Architecture (DS/DMA)(Informatica, Carleton, Sagent)

1998 Executive Information Systems, Inc.

Page 8: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

8

Architectures forManaging DSS Integration (Two)

8 Distributed Data Warehouse/Data MartArchitecture (DDW/DMA) (Sybase, Platinum (HP)Intelligent Warehouse)

8 Distributed Knowledge Management Architecture(DKMA) (The sole vendor targeting this architectureis DAMAN Consulting)

8 There are variations of each architectureincorporating an ODS

1998 Executive Information Systems, Inc.

Page 9: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

9

Top-Down Architecture

8 The first Data Warehousing Systems architecture

8 Begins with Extraction, Transformation,Migration, and Loading (ETML) process

8 Establishes the data warehouse first, along withcentralized metadata repository

8 Data Marts are constituted from extracted andsummarized data warehouse data and metadata

8 The Data Warehouse has atomic layer anddetailed historical data

1998 Executive Information Systems, Inc.

Page 10: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

10

Top-Down Architecture (Two)

8 The Data Warehouse uses Normalized E-RModels

8 Data Marts have highly and lightly summarizeddata

8 Integration is automatic as long as the disciplineof constituting data marts as subsets of the datawarehouse is maintained

8 Tools exist to generate data marts from the datawarehouse “by pushing a button”

1998 Executive Information Systems, Inc.

Page 11: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

11

DataMarts A . . .N

LegacyApplications and

Data Sources

DataWarehouse

Data StagingArea

Extraction,Transformation, &Migration

Figure One -- Top-Down Architecture

Metadata Stores in DW and DMs

1998 Executive Information Systems, Inc.

Page 12: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

12

Bottom-Up Architecture

8 The second Data Warehousing Systemsarchitecture

8 Became popular because the Top-downarchitecture took too long to implement, was oftenpolitically unacceptable, and was too expensive

8 Begins with ETML for one or more Data Marts

8 Requires no common data staging area

8 Uses Dimensional Data models for Data Marts

8 Uses Atomic, lightly summarized, and highlysummarized data

1998 Executive Information Systems, Inc.

Page 13: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

13

Bottom-Up Architecture (Two)

8 Provides no common metadata

8 Constructs thedata warehouse incrementally overtime from data marts

8 Adopted initially by second generation toolvendors Informatica, Sagent, and Ardent

8 Met expectations in building data marts, but soonwas perceived as unacceptable for the long termbecause lacking common metadata, it is difficultto construct the data warehouse, and it also leadsto new “stovepipes” or”legamarts” over time.

1998 Executive Information Systems, Inc.

Page 14: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

14

1998 Executive Information Systems, Inc.

Figure Two -- Bottom-Up Architecture

DataMarts A . . .N

LegacyApplications and

Data Sources

DataWarehouse

Data StagingAreas

Extraction,Transformation, &Migration

Metadata Stores in DW

and DMs

Page 15: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

15

Enterprise DataMart Architecture (EDMA)

8 A response of the Bottom-up supporters to“legamart” argument

8 Begins with ETML for one or more data marts

8 Establishes a common staging area called aDynamic Data Store (DDS) for ET results,including a common or global metadata repository

8 Constructs EDMA before beginning data marts

8 EDMA includes enterprise subject areas, andcommon dimensions, metrics, business rules,sources, all represented in a logically common (butnot necessarily physically centralized) metadatarepository

1998 Executive Information Systems, Inc.

Page 16: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

16

Enterprise DataMart Architecture (EDMA) (Two)

8 Uses Dimensional Data Models for Data Marts

8 Develops the data warehouse and data marts fromthe metadata repository, data marts and the DDSusing an incremental approach

8 Informatica’s PowerCenter Tool was the first toimplement this architecture. Informatica supportsmetadata management through monitoring andreporting mechanisms, not through an automatedprocess. Informatica makes some use of ObjectTechnology. Carleton is also close to thiscapability.

1998 Executive Information Systems, Inc.

Page 17: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

17

1998 Executive Information Systems, Inc.

Figure Three -- EDM Architecture

DataMarts A . . .N

LegacyApplications and

Data Sources

DataWarehouse

Dynamic DataStore Staging

Area

Extraction,Transformation, &Migration

Shared Metadata Repository Layer

Application Servers

Page 18: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

18

Data Stage/Data MartArchitecture (DS/DMA)

8 The same as EDMA with the important exceptionthat no physical enterprise-wide data warehouse isimplemented

8 Instead, the data warehouse is viewed as theconjunction of the data marts in the context of anEDMA-like metadata repository

8 The repository provides a common view ofenterprise DSS resources, but not necessarily anenterprise-wide view, because there is no

1998 Executive Information Systems, Inc.

Page 19: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

19

Data Stage/Data MartArchitecture (DS/DMA) (Two)

8 guarantee that the conjunction of data marts willprovide access to enterprise-wide global attributes,as would a data warehouse

8 Leading tool providers are again Informatica andCarleton, and also Sagent which is advocatingDS/DMA in association with Ralph Kimball

1998 Executive Information Systems, Inc.

Page 20: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

20

1998 Executive Information Systems, Inc.

Figure Four -- DS/DM Architecture

DataMarts A . . .N

LegacyApplications and

Data Sources

The Data Warehouseis the Conjunction of the Data Marts

Dynamic Data Store Staging

Area

Extraction,Transformation, &Migration

Shared Metadata Layer

Application Servers

Page 21: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

21

Distributed Data Warehouse/DataMart Architecture (DDW/DMA)

8 Again similar to EDMA. Also:

8 Provides common view of metadata across theenterprise

8 Provides a logical database layer mapping aunified logical data model to physical tables in thevarious data marts and the data warehouse

8 Provides transparent querying of a unified logicaldatabase across data marts along with caching andjoining services.

1998 Executive Information Systems, Inc.

Page 22: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

22

Distributed Data Warehouse/DataMart Architecture (DDW/DMA) (Two)

8 Leading tool providers are Informatica, Carleton,Sybase Adaptive Server, and HP (now Platinum)Intelligent Warehouse

8 These tools (except IW) are all offered as part ofSybase’s Warehouse Studio

8 This is the most adaptable of the architecturesdiscussed to this point, but it still reflects thelimitations of the relational viewpoint when itcomes to handling objects and processes, and itstill doesn’t support distributed and automatedchange capture and management

1998 Executive Information Systems, Inc.

Page 23: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

23

1998 Executive Information Systems, Inc.Figure Five -- DDW/DM Architecture

DataMarts A . . .N

LegacyApplications and

Data Sources

The Logical/Physical Data

Warehouse Layer Maps a Unified

Data Model to the Physical Tables

in the Data Warehouse and

Data MartsDynamic Data Store Staging

Area

Extraction,Transformation, &Migration

Shared Metadata Layer

Application Servers

Page 24: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

24

Distributed KnowledgeManagement Architecture (DKMA)

8 An evolving O-O/Component-based architecture

8 Top - Down and Bottom-Up architectures may beviewed as two-tier architectures utilizing clientsand local or remote databases

8 EDMA, DS/DMA, and DDW/DMA may beviewed as adding Middleware and Tuple layers toearlier architectures to provide the capability tomanage warehouse systems integration throughunified logical views, monitoring, reporting, andintentional DBA maintenance activity. But this

1998 Executive Information Systems, Inc.

Page 25: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

25

Distributed KnowledgeManagement Architecture (DKMA) (Two)

8 form of management still doesn’t provideautomatic feedback of changes in one componentto others

8 DKMA may be viewed as adding an object layerto EDMA or to DDW/DMA to provide integrationthrough automated change capture andmanagement

1998 Executive Information Systems, Inc.

Page 26: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

26

Distributed KnowledgeManagement Architecture (DKMA) (Three)

8 The object layer contains process distributionservices, an in-memory active, object model, andconnectivity to a variety of data store andapplication types. The layer requires anarchitectural component called an ActiveKnowledge Manager (AKM).

1998 Executive Information Systems, Inc.

Page 27: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

27

1998 Executive Information Systems, Inc.Figure Six -- DKM Architecture

DataMarts A . . .N

LegacyApplications and

Data Sources

The Logical/Physical Data

Warehouse Layer Maps a Unified

Data Model to the Physical Tables

in the Data Warehouse and

Data Marts Dynamic Data Store Staging

AreaExtraction,

Transformation, &Migration

Shared Object Layer with AKM

Process Control ServicesActive In-Memory Object Model

Connectivity Services

PersistentObject Store

Application Servers

Page 28: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

28

DKM Architectureand the AKM

8 An AKM provides Process Control Services, anObject Model of the Distributed KnowledgeManagement System (DKMS) (the systemcorresponding to the DKM architecture), andconnectivity to all enterprise information, datastores, and applications8Process Control Services:

8In memory proactive object state management andsynchronization across distributed objects

8Component management

8Workflow management

8Transactional multithreading 1998 Executive Information Systems, Inc.

Page 29: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

29

DKM Architectureand the AKM (Two)

8In-memory Active Object Model/Persistent ObjectStore is characterized by:8Event-driven behavior

8DKMS-wide model with shared representation

8Declarative business rules

8Caching along with partial instantiation of objects

8A Persistent Object Store for the AKM

8Reflexive Objects

8Connectivity Services should have:8Language APIs: C, C++, Java, CORBA, COM

8Databases: Relational, ODBC, OODBMS,hierarchical, network, flat file, etc.

1998 Executive Information Systems, Inc.

Page 30: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

30

DKM Architectureand the AKM (Three)

8Wrapper connectivity for application software:custom, CORBA, or COM-based.

8Applications include all the categories mentioned inthe earlier discussion of the Dynamic Integrationproblem, whether these are mainframe, server, ordesktop - based.

8 The DKM Architecture and the AKM provide thesolution to the Dynamic Integration Problem,because only the DKMA among the precedingarchitectures supports distributed proactivemonitoring and management of change in the webof data warehouse, data mart, web information

1998 Executive Information Systems, Inc.

Page 31: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

31

DKM Architectureand the AKM (Four)

8 servers, component transaction servers, datamining servers, ETML servers, other applicationservers, and front-end applications comprisingtoday’s Enterprise DSS/Data WarehousingSystem.

1998 Executive Information Systems, Inc.

Page 32: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

32

ODS Variations

8 Each of the architectures covered may vary withthe addition of an Operational Data Store (ODS)

8 According to Inmon: “An ODS is a collection ofdata containing detailed data for the purpose ofsatisfying the collective, integrated operationalneeds of the corporation . . . The ODS is:8subject-oriented,

8 integrated,

8volatile,

8current-valued,

8detailed.”

1998 Executive Information Systems, Inc.

Page 33: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

33

ODS Variations (Two)

8 The ODS is like a data warehouse in its first twocharacteristics, but it is like an OLTP system in itslast three characteristics. Its purpose is to supportoperational, tactical decisions

8 The workload of an ODS involves four kinds ofprocessing: loading data, updating, accessprocessing, and DSS-style analysis across manyrecords.

8 The four types of ODS processing are the sourceof difficulties in optimizing ODS processing. It isdifficult to optimize performance over all fourtypes.

1998 Executive Information Systems, Inc.

Page 34: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

34

ODS Variations (Three)

8 Look at the above architectures in relation to theODS. It is clear that an architecture that willsupport both DSS and OLTP-style processing isneeded in order to optimally integrate the ODSinto the broader data warehousing architecture. Inparticular,8process control services will be very important for

the OLTP-style of processing we find in the ODS.

8Also, distribution of ODS objects across multipleservers will help ODS performance.

1998 Executive Information Systems, Inc.

Page 35: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

35

ODS Variations (Four)

8Finally, in-memory processing in distributedAKMs can do much to upgrade performance in adistributed ODS.

8 Of course, only one of the above architecturescan provide these capabilities for the ODS: theDKM Architecture.

1998 Executive Information Systems, Inc.

Page 36: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

36

DKM Architectureand Data Mining

8 A key emerging capability in DKMS and datawarehousing systems is Knowledge Discovery inDatabases (KDD) or Data Mining.

8 The key mechanism for KDD is the data miningserver.

8 Here are some difficulties with current data miningserver products:8It’s difficult to incorporate new data mining

algorithms, and therefore keep pace with newdevelopments coming out of the research world;

1998 Executive Information Systems, Inc.

Page 37: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

37

DKM Architectureand Data Mining (Two)

8Many products require that data must be transportedto proprietary data stores before data mining canoccur;

8Models produced by the data mining algorithms arenot freely available to power users unless they usethe data mining tool itself,

8It is difficult to incorporate validation criteria notinitially incorporated in the data mining tool intothe KDD process,

8There are few “open architecture” commercial datamining tools.

1998 Executive Information Systems, Inc.

Page 38: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

38

DKM Architectureand Data Mining (Three)

8 To solve these problems a product class called AnAnalytical Data Mining Workbench (ADMW)should be developed.

8 The ADMW needs:8Easy and convenient encapsulation of new

algorithms into object model classes;

8Capability to mine data from any data source in theenterprise;

8Incorporation of analytical models into an objectmodel repository;

8A modifiable validation model, 1998 Executive Information Systems, Inc.

Page 39: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

39

DKM Architectureand Data Mining (Four)

8Integration of legacy data mining applications withthe ADMW.

8 An ADMW with these capabilities would meet allof the difficulties specified above.

8 The DKM Architecture can help in developing theabove because:8New algorithms can be encapsulated in objects

through the “wrapping” capabilities of the AKM;

8Data can be brought into the AKM’s in-memoryobject model for data mining without relocating itfrom its data store (“chunks,” partial instantiation),

1998 Executive Information Systems, Inc.

Page 40: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

40

DKM Architectureand Data Mining (Five)

8Data mining can be performed by executing theanalytical models in memory on data chunks andpartially instantiated objects;

8Analytical Models produced by an AKM -basedapplication would be placed in an object modelrepository where they can be accessed by PowerUsers;

8Customized validity criteria could be added bymodifying the validation model in the repository,because the validation model is just another objectwhose attributes and methods can be modified ;

1998 Executive Information Systems, Inc.

Page 41: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

41

DKM Architectureand Data Mining (Six)

8Legacy data mining applications could beintegrated using AKM connectivity services whichwould “wrap” them.

8 When viewing the above, keep in mind that thereis no ADMW with the above capabilities atpresent. Data Mining is a rapidly growing field,but the market niche represented by the ADMW isempty.

8 On the other hand, there are software tools that canbe used as a foundation to rapidly develop theADMW as a facility within the AKM.

1998 Executive Information Systems, Inc.

Page 42: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

42

DKM Architectureand Software Tools

8 To implement DKM Architecture in a DKMS youneed the full range of tools now used to create datawarehousing systems. In addition though, you needadditional tools for the AKM component(including the ADMW facility and the ability tointegrate the ODS into the DKMS). These include:8An object modeling RAD environment providing

extensive process control services and connectivity( e.g. DAMAN's InfoManager, TemplateSoftware’s Enterprise Integration Template (EIT),Forte, a combination of Ibex's DAWN workflowproduct along with its Itasca Active Object

1998 Executive Information Systems, Inc.

Page 43: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

43

DKM Architectureand Software Tools (Two)

8Database, a combination of Rational Rose,Persistence Power-Tier; and Iona’s Orbix)

8Technology for constructing software agents toproactively monitor components of the DKMS (e.g.CA Unicenter TNG, ObjectSpace’s Voyager,Persistence Power-Tier, DAMAN’s InfoManager)

8An OODBMS to serve as a persistent objectrepository for the AKM component (ObjectStore,Objectivity/DB. Jasmine, Versant, Itasca)

1998 Executive Information Systems, Inc.

Page 44: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

44

Page 45: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

45

Back-Up Slides

4Distributed Knowledge Management Systems (DKMS

4Why DKMS?

4What is the Knowledge Management System (KMS)?

4The Knowledge Base and Knowledge

4The Knowledge Management Process and KnowledgeManagement

4Data, Information, Knowledge, and Wisdom

4Organizational Knowledge

Page 46: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

46

Distributed KnowledgeManagement Systems (DKMS)

8 A DKMS is a system that manages the integration of distributed objectsinto a functioning whole producing, maintaining, and enhancing abusiness knowledge base.

8 A business knowledge base is the set of data, validated models,metamodels, and software used for manipulating these, pertaining tothe enterprise, produced either by using a DKMS, or imported fromother sources upon creation of a DKMS. A DKMS, in this view,requires a knowledge base to begin operation. But it enhances its ownknowledge base with the passage of time because it is a self-correctingsystem, subject to testing against experience.

8 The DKMS must not only manage data, but all of the objects, objectmodels, process models, use case models, object interaction models,and dynamic models, used to process data and to interpret it to producea business knowledge base. It is because of its role in managing andprocessing data, objects, and models to produce, enhance, and maintaina knowledge base that the term Distributed Knowledge ManagementSystem is so appropriate.

1998 Executive Information Systems, Inc.

Page 47: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

47

Why DKMS?

8Other reasons for adopting the term DKMS include:

8business knowledge production and management iswhat business intelligence is all about;

8DKMS plays off DBMS, and therefore capitalizes on afamilar term while favorably contrasting with it, i.e.knowledge management is clearly better than mere datamanagement;

8DKMS also highlights the point that data is notknowledge, but only a part of it;

8“DKMS” is a product/results-oriented name likely toappeal to business decision makers (that is, they getvaluable and valid knowledge that they can use to gaincontrol and produce results);

1998 Executive Information Systems, Inc.

Page 48: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

48

What is the KnowledgeManagement System (KMS)?

8 The Knowledge Management System (KMS) is the on-going, persistentinteraction among agents within a system that produces, maintains, andenhances the system's knowledge base. This definition is meant toapply to any intelligent, adaptive system composed of interactingagents.

8 An agent is a purposive, self-directed object.

8 Knowledge Base will be defined in the next section.

8 In saying that a system produces knowledge we are saying that it (a)gathers information and (b) compares conceptual formulationsdescribing and evaluating its experience, with its goals, objectives,expectations or past formulations of descriptions, or evaluations.

8 Further, this comparison is conducted with reference to validationcriteria. Through use of such criteria, intelligent systems distinguishcompeting descriptions and evaluations in terms of closeness to thetruth, closeness to the legitimate, and closeness to the beautiful.

1998 Executive Information Systems, Inc.

Page 49: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

49

What is the KnowledgeManagement System (KMS)? (Two)

8 In saying that a system maintains knowledge we are saying that itcontinues to evaluate its knowledge base against new information bysubjecting the knowledge base to continuous testing against itsvalidation criteria. We are also saying that to maintain its knowledge, amore complex system must ensure both the continued dissemination ofits currently validated knowledge base, and continued socialization ofintelligent agents in the use and content of its knowledge base.

8 Finally, in saying that a system enhances its knowledge base, we aresaying that it adds new propositions and new models to its knowledgebase, and also simplifies and increases the explanatory and predictivepower of its older propositions and models. That is, one of thefunctions of the KMS is to provide for the growth of knowledge.

1998 Executive Information Systems, Inc.

Page 50: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

50

The Knowledge Base and Knowledge

8 A system's knowledge base is: the set of remembered data; validatedpropositions and models (along with metadata related to their testing);refuted propositions and models (along with metadata related to theirrefutation); metamodels; and (perhaps, if the system produces such anartifact) software used for manipulating these, pertaining to the systemand produced by it.

8 A knowledge management system, in this view, requires a knowledgebase to begin operation. But it enhances its own knowledge base withthe passage of time because it is a self-correcting system, and subjectsits knowledge base to testing against experience.

8 Finally, the emphasis on a system's knowledge base, rather than itsknowledge, recognizes that an identification of knowledge as individualconceptions, propositions, or models is inconsistent with the reality thatacceptance of a piece of information into a system's body of knowledgeis dependent on the background knowledge already within theknowledge base. This background knowledge is used to filter andinterpret the information being evaluated.

1998 Executive Information Systems, Inc.

Page 51: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

51

The Knowledge Base and Knowledge (Two)

8 This definition of knowledge base contrasts with a popular definition ofknowledge as "justified, true belief." The definition does agree with thenecessity of justification as a necessary condition for knowledge; but itinsists that justification be specific to the validation criteria used by asystem to evaluate its descriptions and evaluations. The definition alsoagrees that knowledge is a particular kind of belief, provided that beliefextends beyond cognition alone, to evaluation.

8 The biggest discrepancy with the popular definition is in not requiringthat justified beliefs be "true." Truth can be used as a regulating idealby a system producing descriptive knowledge. "Right" can be used as aregulating ideal by a system producing evaluative or normativeknowledge. But the system in question can never say for sure that aproposition or a model within its knowledge base is "true," or "right;"but only that it has survived refutation by experience better than itscompetitors, and therefore that the system "believes" it is true or right.

1998 Executive Information Systems, Inc.

Page 52: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

52

The Knowledge Base and Knowledge (Three)

8 So instead of knowledge as "true, justified belief," the position taken hereis that knowledge equals justified belief that some conceptual formulation,fact, or evaluation, is true or right as the case may be.

8 In a very real sense, a system's knowledge is the analytical network ofpropositions and models constituting the knowledge base. It istherefore, just for convenience, that one may refer to a particularproposition or model as something a system "knows," because it knowsthat "something," only if one assumes that numerous unspecifiedbackground propositions and models are also known by it.

1998 Executive Information Systems, Inc.

Page 53: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

53

The Knowledge Management Processand Knowledge Management

8The Knowledge Management Process (KMP) is an on-going persistent interaction among human-based agentswho aim at integrating all of the various agents,components, and activities of the knowledge managementsystem into a planned, directed process producing,maintaining and enhancing the knowledge base of theKMS.

8Knowledge Management is the human activity within theKMP aimed at creating and maintaining this integration,and its associated planned, directed process.

1998 Executive Information Systems, Inc.

Page 54: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

54

The Knowledge Management Processand Knowledge Management (Two)

8 A good way to look at the human activity called knowledgemanagement is through the concept of the Use Case. In a use case ahuman-based agent, within the KMS, called an actor, participates in theKMP to get an outcome from the KMS that has value for the actor.The KMP can be represented as a set of Business Process Use Caseseach classified within one of four business sub-process categories:planning, acting, monitoring, and evaluating. A way of decomposingknowledge management activity then, is in terms of the use cases thatconstitute it.

8 The set of all use cases aimed at creating and maintaining theintegrated, planned, directed process producing, enhancing andmaintaining the KMS knowledge base, is an alternative characterizationof knowledge management. The set of these use cases represents all ofthe organizational knowledge management activity of the actorsmaking use of the KMS through the KMP. In other words, the set ofuse cases is what we mean by knowledge management in a humansystem.

1998 Executive Information Systems, Inc.

Page 55: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

55

Data, Information,Knowledge, and Wisdom

8 What is the difference between data, information, knowledge, andwisdom?

8 To begin with, organizational data, information, knowledge, andwisdom, all emerge from the social process of an organization, and arenot private. In defining them, we are not trying to formulate definitionsthat will elucidate the nature of personal data, information, knowledge,or wisdom. Instead, to use a word that used to be more popular indiscourse than it is at present, we are trying to specify intersubjectiveconstructs and to provide metrics for them.

8 A datum is the value of an observable, measurable or calculableattribute. Data is more than one such attribute value. Is a datum (or isdata) information? Yes, information is provided by a datum, or by data,but only because data is always specified in some conceptual context.At a minimum, the context must include the class to which the attributebelongs, the object which is a member of that class, some ideas aboutobject operations or behavior, and relationships to other objects andclasses.

1998 Executive Information Systems, Inc.

Page 56: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

56

Data, Information,Knowledge, and Wisdom (Two)

8 Data alone and in the abstract therefore, does not provide information.Rather, information, in general terms, is data plus conceptualcommitments and interpretations. Information is data extracted, filteredor formatted in some way (but keep in mind that data is alwaysextracted filtered, or formatted in some way).

8 Knowledge is a subset of information. But it is a subset that has beenextracted, filtered, or formatted in a very special way. Morespecifically, the information we call knowledge is information that hasbeen subjected to, and passed tests of validation. Common senseknowledge is information that has been validated by common senseexperience. Scientific knowledge is information (hypotheses andtheories) validated by the rules and tests applied to it by some scientificcommunity.

8 Wisdom, lastly, has a more active component than data, information, orknowledge. It is the application of knowledge expressed in principles toarrive at prudent, sagacious decisions about conflict situations.

1998 Executive Information Systems, Inc.

Page 57: Executive Information Systems, Inc. Architectural ...8and various front-end OLAP and reporting tools ... capability that drives architectural evolution in the DW system. 1998 Executive

57

Organizational Knowledge

8 Organizational knowledge in terms of this framework is informationvalidated by the rules and tests of the organization seeking knowledge.The quality of its knowledge then, will be largely dependent on thetendency of its validation rules and tests to produce knowledge thatimproves organizational performance (the organization’s version ofobjective knowledge).

8 From the viewpoint of the definition given of organizationalknowledge, what is an organization doing when it validates informationto produce knowledge? The validation process is an essential aspect ofthe broader organizational learning process, and that validation is aform of learning. So, though knowledge is a product and not a processderived from learning, knowledge validation (validation of informationto admit it into the knowledge base) is certainly closely tied to learning,and depending on the definition of organizational learning, may beviewed as derived from it.

1998 Executive Information Systems, Inc.