Information & System Quality Considering and assuring quality dimensions in architecture design...
-
date post
20-Dec-2015 -
Category
Documents
-
view
219 -
download
3
Transcript of Information & System Quality Considering and assuring quality dimensions in architecture design...
Information & System Quality
Considering and assuring quality dimensions in architecture design
"Drowning in data, yet starved of information"(Ruth Stanat, 1990, in 'The Intelligent Corporation’ )
Ir. Nitesh Bharosa | [email protected]
11-02-2010
2
Who am I?
Nitesh Bharosa• PHD candidate at the ICT Section (finishing in January
2011)• M.Sc. in Systems Engineering, Policy Analysis and
Management Thesis: Enterprise Architecture at SiemensResearch interest
• information & system quality• orchestration & coordination• enterprise-architecture, SOA, SAAS, • public safety and disaster management
•Courses: • SPM3410 Web information Systems and Management• SPM4341 Design of Innovative ICT-infrastructures and
services,• guest lectures e-business and management of
technology
3
Today’s goals
• Understand the concepts of information and system quality in multi-actor environments
• Be able to distinguish multiple information quality dimensions
• Be able to distinguish multiple systems quality dimensions
• Understand principles for assuring information and system quality
• Introduction to “Master of Disaster Game”
4
Further reading
• Strong, Lee & Wang. (1997). Data quality in context. Communications of the ACM.
• Nelson et al (2002). Antecedents of information and system quality. Journal of Management Information Systems.
• Bharosa, N., et al (2009). Identifying and confirming information and system quality requirements for multi-agency disaster management. In the ISCRAM 2009 proceedings.
5
Agenda
1. Background and relevance2. Concepts and definitions3. Hurdles for IQ and SQ in practice4. Complex multi actor case: Disaster
management5. How do we assure information and system
quality in the architecture?6. Summary and conclusions
When was the last time you were encountered
with wrong information?
7
Information Systems Success theory*
*Delone & Mclean (1992). Information Systems Success: the quest for the dependent variable. Information Systems Research, 3(1), pp.60-95
Information Quality
System Quality
8
Relevance of poor IQ for the typical enterprise*
• Operational Impacts:• Lowered customer satisfaction• Increased cost: 8–12% of revenue in the few, carefully studied cases• For service organizations, 40–60% of expense• Lowered employee satisfaction
• Typical Impacts:• Poorer decision making: Poorer decisions that take longer to make• More difficult to implement data warehouses• More difficult to reengineer• Increased organizational mistrust
• Strategic Impacts:• More difficult to set strategy• More difficult to execute strategy• Contribute to issues of data ownership• Compromise ability to align organizations
*based on Redman (2002)
9
What is information quality?
10
The concept of quality in Information systems
• Quality is not a new concept in information systems management and research
• What is ‘new’ is the explosion in the quantity of information and the increasing reliance of most segments of society on that information
• Challenges: defining and improving quality for a specific context
• Information systems researchers have attempted to define data quality, information quality software quality, system quality, documentation quality, service quality, web quality and global information systems quality
11
Some definitions for IQ
• Quality information is information that meets specifications or requirements (Khan & Strong, 1999)
• IQ is the characteristic of information to meet the functional, technical, cognitive, and aesthetic requirements of information producers, administrators, consumers, and experts (Eppler, 2003)
• Information of high IQ is fit for use by information consumers (Huang, Lee, Wang, 1999, p. 43)
• IQ as set of dimensions describing the quality of the information produced by the information system (Delone & Mclean, 1992).
• Quality of information can be defined as a difference between the required information (determined by a goal) and the obtained information (Gerkes, 1997)
12
IQ Frameworks * 1
Information as a product Information as a process
· usefulness· comprehensibility· relevancy· completeness· adequate representation· coherence· clarity
· trustworthiness· accessibility· objectivity· credibility· interactivity (feedback)
*Lesca & Lesca (1995)
13
IQ Frameworks * 2
Perspective Criteria
Content relevance, obtainability, clarity of definition
Scope Comprehensiveness, essentialness
Level of detail Attribute granularity, precision of domains
Composition Naturalness, identifiability, homogeneity, minimum unnecessary redundancy
View Consistency
Semantic consistency, structural consistency Conceptual View
Reaction to change
Robustness, flexibility
Values Accuracy, completeness, consistency, currency/ cycle time
*Redman (1996) Data Quality for the information age
14
IQ Frameworks* 3a
Category Dimension Meaning
Intrinsicdataquality
Accuracy The extent to which information represents the underlying reality.
Objectivity The extent to which information is unbiased, unprejudiced and impartial
Believability The extent to which information is regarded as true and credible.
Reputation The extent to which information is highly regarded in terms of its source or content
Accessibility dataquality
Accessibility The extent to which information is available, or easily and quickly retrievable.
Access Security
The extent to which access to information is restricted appropriately to maintain its security
*Strong, D. M., Lee, Y. W., & Wang, R. Y. 1997. Data Quality in Context. Communications of the ACM, 40(5): pp.103-110.
15
IQ frameworks* 3bCategory Dimension MeaningContextual dataquality
Relevancy The extent to which information is applicable and helpful for the task at hand
Value added The extent to which information is beneficial and provides advantages from its use
Timeliness The extent to which information is sufficiently up to date for the task at hand
Completeness The extent to which information is not missing and is of sufficient bread and depth for the tasks at hand
Appropriate amount of data
The extent to which the volume information is appropriate for the tasks at hand
Representationaldataquality
Interpretability The extent to which information is appropriate languages, symbols and units and the definitions are clear
Concise representation
The extent to which information is composedly represented
Consistentrepresentation
The extent to which information is presented in the same format
Understandability The extent to which information is easy comprehended
*Strong, D. M., Lee, Y. W., & Wang, R. Y. 1997. Data Quality in Context. Communications of the ACM, 40(5): pp.103-110.
16
Discussion:Is there a difference between data quality and information quality?An what about knowledge and
wisdom?
17
Transitions from data to wisdom
(raw) Data
Information
Knowledge
Processing (use of information systems)
Internalization over time (human processing, can be tacit)
Volume
Complexity of quality management
Intelligence
Based on level of understanding & experience
18
19
Data, Information, Knowledge and Wisdom*
• Data is an discrete, unitary, and indivisible element which conveys a single value. Data serves as the basis for computation and reasoning to be executed
• Information is an aggregate of one or more data elements with certain established relationships, and it has the ability to convey a single, meaningful message
• Knowledge is a large-scale selective combination or union of related pieces of information accumulated over a prolonged period of time, and it can be viewed as a discipline area
• Wisdom is the new knowledge subset created when the deductive ability acquired by a person after attaining a sufficient level of understanding of a knowledge area is executed
*Adapted from Liang (1994)
20
Data to information processing*
Al-Hakim (2007) Information Quality Function Deployment
21
Subjective and context dependent nature of information
• “Perfect” IQ, is difficult, if not impossible, to achieve
• but neither is it necessary!• If users of the data feel that its quality, which
can be described by such attributes as accuracy, completeness and timeliness, is sufficient for their needs, then, from their perspective, at least, the quality of the information available to them is fine
• Hence we need a clear understanding of user processes and their information needs in specific context
22
What is system quality?
23
System Quality
• Defined as: the quality of the information system (as producing system) and not of the information (as product) (Delone & McLean, 1992)
• Also not a ‘new’ concept in information systems• However, this concept has received less formal
and coherent treatment than information quality
• Trend: information systems are becoming more than just single software applications
• SQ is also an antecedent for information system success
24
Examples of poor system quality 1
SQ dimension
Example
Accessibility The 9/11 case shows that access to data across agency lines also needs to be improved to support interagency coordination (Comfort & Kapucu, 2006). “In some cases, needed information existed but was not accessible” (Dawes et al., 2004)
Response time As much of the information needed during the response is time sensitive a low response time is necessary (Board on Natural Disasters, 1999). In case of emergencies, time is of the essence—every moment of delay can significantly reduce an accident victim’s chances of survival (Horan & Schooley, 2007) underlining the need for low response times
Reliability “…responding to disaster situations, where every second counts, requires reliable, dedicated equipment. Experience has shown that these systems are often the most unreliable during critical incidents when public demand overwhelms the systems” (National Research Council, 2007)
25
Examples of poor system quality 2
SQ dimension
Example
Interoperability
“…given the number of organizations that must come together to cope with a major disaster, the interoperability of communications and other IT systems is often cited as a major concern” (National Research Council, 2007)
Integration The need for integration intensifies as the number of organizations engaged in response operations increases and the range of problems they confront widens (Comfort & Kapucu, 2006)
Flexibility “A catastrophic incident has unique dimensions/ characteristics requiring that response plans/strategies be flexible enough to effectively address emerging needs and requirements” (National Research Council, 2007)
26
Complex multi-actor systems
• Examples include supply chains, value networks traffic systems and crisis management networks
• In such systems, intra- and inter organizational information flows need to be coordinated in order to achieve goals: high interdependency
• Information systems play in critical role in the coordination process
• Multiple echelons of coordination: strategic, tactical and operational
• Actors operate in a complex, dynamic and unpredictable task environment
27
IQ & SQ issues during disaster response
• Chernobyl (1986)• Herculus (1999)• Enschede (2000)• New York (2001)• Singapore (2003)• Tsunami (2004)• Schiphol (2006)• Delft (2008)• …
Disaster Management
28
Complexity: heterogeneous actors and systems during 9/11 response
*source: Comfort, L. (2002), ‘‘Rethinking Security: Organizational Fragility in Extreme Events,’’ Public Administration Review 62, Special Issue (September), 98–107
29
Information flows in the Netherlands Strategic Echelon
Tactical Echelon
Operational Echelon
Em
ergency Control room
Practice 1: distributed teams
Manual situation
report generation
Practice 2: several information types, formats, sources and technologies
32
Examples of poor IQ during disasters
IQ dimension Example
Completeness In the response to the 2004 Tsunami, “mostly, the information is incomplete, yet conclusions must be drawn immediately” (Samarajiva, 2005). “During Katrina, the federal government lacked the timely, accurate, and relevant ground-truth information necessary to evaluate which critical infrastructures were damaged, inoperative, or both” (Townsend et al, 2006)
Correctness Firefighters rushing to the Shiphol Detention Complex received incorrect information about the open gates to the area and were delayed in finding the right gate (Van Vollehoven et al, 2006)
Relevancy When police helicopters observed that one of the Twin Towers was going to collapse, they immediately requested all police officers leave the building. Despite that this information was also relevant for firefighters and ambulance services, they had never received this information and as a result, almost 400 of them died
33
Examples of poor SQ during disastersSQ
dimensionExample
Accessibility The 9/11 case shows that access to data across agency lines also needs to be improved to support interagency coordination (Comfort & Kapucu, 2006). “In some cases, needed information existed but was not accessible” (Dawes, et al., 2004).
Response time If there was a comprehensive plan to quickly communicate critical information to the emergency responders and area residents who needed it, the mixed messages from Federal, State, and local officials on the reentry into New Orleans could have been avoided (Townsend et al, 2006).
Flexibility “A catastrophic incident has unique characteristics requiring that response systems be flexible enough to effectively address emerging needs and requirements” (National Research Council, 2007). “The lack of such capacity at the regional level (incl. municipalities, counties, districts, nonprofit and private institutions), was evident in the effort to mobilize response to the 9/11 events” (Comfort & Kapucu, 2006).
34
Main Challenge: Assuring IQ and SQ in MAS
+
?+
Information Quality
SystemQuality
35
Some generic steps in the assurance process
1. Understand the stakeholder goals and information needs2. Model the process and information flows3. Define clear IQ and SQ measurement instruments4. Analyze hurdles for IQ and SQ (symptoms) on the
various architectural layers (i.e., via observations and interviews)
5. Synthesize principles for assuring IQ and SQ6. Implement and evaluate principles (i.e., prototyping,
gaming simulation)7. Train awareness: information as a product8. Capture feedback and start over again (continuous
process)
36
1. Stakeholder Analysis
• Consumers/clients• Process architects• Database architects• Data suppliers• Application architects• Communication trainers• Programmers• Managers (CIO, CTO etc)• Auditors etc
37
2. Process and information flow modelingEmergency Control Room (ECR)
Commando Place Incident
(CoPI)
Municipal Crisis Center (MCC)
Field Workers
Go to Stations
Get acquainted and read material
Get acquainted and read material
Get acquainted and read material
Get acquainted and read material
Send Emergency Messages
Read Message and Broadcast
thru DIOSPDA
Read Emergency Message on
Beamer
Read Emergency Message on
Laptop
Read Emergency Message on
Laptop
Complete SITRAP by
filling in DIOS
Complete SITRAP by
filling in DIOS
Exchange Info Requests with
Field
Receive CoPI Information Requests
Reply on Mail and store
information
IM: Interpret and react on DIOS output
IM: Interpret and react on DIOS output
Give Press Conference
Go to Info Point with information
requests
Process Information Requests
Send Emergency Messages
38
3a. IQ and SQ measurement
• Context dependent• Multidimensional constructs• Subjective: dependent on the user judgment• So, how do we measure IQ and SQ?• Need for multiple instruments
• Questionnaires (paper or online)• Observations• Interviews• Focus groups• Gaming
39
3b. IQ measurement items
Strongly Disagree
NeutralStrongly
Agree
The information I received from others was timely (up-to-date).
1 2 3 4 5 6 7
The information I received from others was correct (free-of-error)
1 2 3 4 5 6 7
The information I received from others was accurate (no missing piece of information)
1 2 3 4 5 6 7
Others provided me with too much information 1 2 3 4 5 6 7
The information I received from others was relevant (directly applicable to my decisions or actions)
1 2 3 4 5 6 7
The information I received from others was consistent (not contradicting to other information)
1 2 3 4 5 6 7
40
3c. SQ measurement itemsStrongly Disagree
NeutralStrongly
Agree
The information system immediately provided the information I requested
1 2 3 4 5 6 7
I was able to obtain all the information I needed using the information system
1 2 3 4 5 6 7
The information system provided me with relevant information
1 2 3 4 5 6 7
The information system provided me with contradicting information
1 2 3 4 5 6 7
The response time of the information system was too high (I had to wait too long for the information I requested)
1 2 3 4 5 6 7
The information provided by the information system was in an easily understandable format (uncomplicated)
1 2 3 4 5 6 7
41
4. Hurdles in MASArchitecture Layers
Typical hurdles
Stakeholder Ownership, isolation from processes, individual processing capability (overload), context and subjective interpretation
Network Fragmentation, the politics of information, incentives to share, security and privacy requirements
Process Event uncertainty, ad-hoc and unprecedented process flows, changing tasks and information needs
Data Multiple databases, large volumes, aggregation, integrating external and internal data, refining data into classified actionable 'chunks'
Technology Heterogeneity, silo’s, incompatible standards, user accessibility, interface to sources, retrieval, reliability (up-time)
42
5a. Strategies to avoid poor IQ and SQ
• Sender or source based strategies• e.g., rules and policies, data cleansing
• Receiver or destination based strategies• e.g., filters, aggregation algorithms
• Mediation or network based strategies• e.g., stewardship and “Information
Orchestration”
43
5b. Conventional source based techniques for IQ improvement
• data cleansing & normalization (Hernadez & Stolfo, 1998),
• data tracking & statistical process control (Redman, 1996),
• data source calculus & algebra (Lee, Bressen, & Madnick, 1998)
• data stewardship (English, 1999)• dimensional gap analysis (Kahn, Strong, & Wang, 2002)• Usually there are four steps involved
1. Profiling and identification of DQ problems2. Reviewing and characterize of expectations (business
rules)3. Instrument development and Measurement4. Solution proposition and implementation
44
5c. Conventional techniques for SQ improvement
• More/better hardware• More/better software• Reduce number of nodes in the information flow• Redundancy (reliability and robustness)• Less forms and procedures in the information
exchange process
45
5d. Limitations of conventional assurance approaches
• More databases and technologies include higher cost and do not solve IQ and SQ problems in coherence
• Assume a “static” data layer• Do not address task environment dynamics and
uncertainty• Reactive, do not include strategies for sensing
and adapting • Need for proactive mechanisms to deal with
dynamic information needs
46
5e. An information orchestration approach
46
Protective principles (e.g., dependency diversification)
Exploitative principles (e.g., proactive sensing)
Corrective principles (e.g., IQ rating)
Information Orchestration
Preemptive principles (e.g., IQ auditing)
Defensive
Offensive Dynamic adjustment strategy
Advance structuring strategy
Before a disaster
During a disaster
47
5f. Advance structuring strategy and principles
• Examples of preemptive principles• Treat information as product not by-product• Organize IQ audits on a regular basis• Assign IQ roles and responsibilities across
organizational units
• Examples of protective principles• Maximize the number of sources for each information
object• Define several information access and manipulation
levels• Strive for loosely coupled application components
48
5g. Dynamic adjustment strategy and principles
• Examples of exploitative principles• Anticipate information needs prior to the occurrence of
events• Exploit multi-channel and technology convergence• Scan the environment for complementary information
• Examples of corrective principles• Maximize the number of feedback opportunities across
the network• Develop policies for ascertaining information needs,
acquiring and managing information throughout its life cycle
• Encourage a sharing culture (data to information transformation by collective interpretation, discussion & expert analysis)
49
6a. Prototyping
6b. Gaming simulation
51
7. Information as product or by-product*
* Source: Lee et al (2006) Journey to data quality
52
IQ Assurance requires trade-offs* between:• security & accessibility: the more secure an
information system is, the less convenient is its access• timeliness & accuracy: the more current a piece of
information has to be, the less time is available to check on its accuracy
• correctness or reliability and timeliness: the faster information has to be delivered to the end-user, the less time is available to check its reliability or correctness
• right amount of information (or scope) and comprehensibility: more detailed information can prevent a fast comprehension, because it becomes difficult “to see the big picture”
• conciseness & right amount (scope) of information: the more detail that is provided, the less concise a piece of information or document is going to be
*based on Eppler (2003)
53
SQ assurance tradeoffs
• Flexibility versus robustness• Accessibility versus security• Security versus interoperability• Reliability versus flexibility• Availability versus cost• Adaptability versus accountability
54
Conclusions for today
• Assuring high IQ and SQ is becoming more important and more problematic
• The hurdles for IQ and SQ are abundant and multi-level• There is no one best (technical) solution for IQ problems,
the solution space covers multiple architecture layers (e.g., organizational, process and technical layers)
• Assuring IQ and SQ is an continuous process and needs to be institutionalized/embodied in the organizational culture
• There are many information quality dimensions and not all are relevant: some tradeoffs need to be made
Questions and
Discussion