Chance Discovery With Data Crystallization - Discovering Unobservable Events_2006

7/28/2019 Chance Discovery With Data Crystallization - Discovering Unobservable Events_2006

1/48

Final Report

Title: Chance Discovery with Data Crystallization

A Basic Research for Discovering Unobservable Events

Contract Number: FA5209-05-P-0259

AFOSR/AOARD Reference Number: AOARD-05-15

AFOSR/AOARD Program Manager: Tae-Woo Park, Ph.D.

Period of Performance: 01 April 2005 - 31 March 2006

Submission Date: 10 May 2006

PI: Yukio Ohsawa /University of Tsukuba3-29-1 Otsuka, Bunkyo-ku, Tokyo


2/48

Report Documentation PageForm Approved

OMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and

maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,

including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington

VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing t o comply with a collection of information if it

does not display a currently valid OMB control number.

1. REPORT DATE

08 AUG 2006

2. REPORT TYPE

Final Report (Technical)

3. DATES COVERED

01-04-2005 to 31-03-2006

4. TITLE AND SUBTITLE

Chance Discovery with Data Crystallization - Discovering Unobservable

Events

5a. CONTRACT NUMBER

FA520905P0259

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Yukio Ohsawa

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

University of Tsukuba,3-29-1 Otsuka, Bunkyo,Tokyo112-0012,Japan,JP,1120012

8. PERFORMING ORGANIZATION

REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

The US Resarch Labolatory, AOARD/AFOSR, Unit 45002, APO, AP,

96337-5002

10. SPONSOR/MONITORS ACRONYM(S)

AOARD/AFOSR

11. SPONSOR/MONITORS REPORT

NUMBER(S)

AOARD-054016

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT

It is only the observable part of the real world that can be presented in data. For such a scattered, i.e., an

incomplete and ill-structured data, data crystallizing aims at presenting the hidden structure by inserting

dummy items corresponding to unobservable, i.e., hidden events, to the given data on past events. The

existence of hidden events and their position in the environment will be visualized as a result of data

crystallizing. This basic method is expected to be applicable for various real world domains to which

chance-discovery methods have been applied. This project aims at developing the process of data

crystallizing, with a new tool extending KeyGraph, based on the process of chance discovery. In the

research, experiments will be made using artificial data obtained from simulating the target of intelligence

analysis, i.e., organized crimes.

15. SUBJECT TERMSData Mining, Chance Discovery

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OFABSTRACT

18. NUMBER

OF PAGES

47

19a. NAME OF

RESPONSIBLE PERSONa. REPORT

unclassified

b. ABSTRACT

unclassified

c. THIS PAGE

unclassified

Standard Form 298 (Rev. 8-98)Prescribed by ANSI Std Z39-18


3/48


4/48

- Marketing, where consumer-behaviors from hidden motivations are dealt with,- Prediction of earthquakes caused by hidden active faults- Hepatitis treatment, where some observation might be missing in the blood test.In studies on chance discovery, we have been working well in finding rare but significant events. Data

crystallizing means to extend chance discovery to the discovery of significant events which have never

occurred in the given data, i.e., from low-frequency to zero-frequency. This means to deal with more uncertain

environment where human may miss important event, than we have been dealing with in data mining or

chance discovery.

A relevant research area to Chance Discovery is Evidence Extraction and Link Discovery (EELD),

where important links of people with other people and with their own actions are to be discovered from

heterogeneous sources of data. The difference between Chance Discovery and EELD, at the time we began this

project, was in the position of human factors in the research approaches. In Chance Discovery, the

visualization techniques such as KeyGraph have been used for clarifying the effect of chances, by enforcing

the users thoughts on scenarios in the real environment. On the other hand, the EELD program mainlycontributed to identifying the most significant links among items more automatically and precisely than human.

After the one year of this successful project, we showed an improvement of the visualization tool reinforcesthe process of chance discovery, and this may be regarded as a new feature of the state of chance discovery.

I expect these two will meet, because the studies in EELD is now oriented to coupling symbolic

expressions of human knowledge with a machine learning system. That is, humans interaction with machine

intelligence is coming to the centers of these two domains. Some studies in EELD, such as data visualization

for decision making, serve bridges between human and machine. In this sense, our methods for data

crystallization is expected to contribute to EELD as well as to chance discovery.

Relation to the goal

The sphere of real world applications linked from this basic research is expected to include intelligence

analysis aiming to arrest unknown leaders, development of new (unknown) products, aiding corporate

behaviors by detecting unknown interest of employees, etc. We successfully accomplished to show the

potential ability of our methods to solve these new problems, by applying to toy (simulated) and real problems

corresponding to small-size version of these up-to-date problems.

(5) Personnel Supported: List the professional personnel supported by the contract and/or the personnel who

participated significantly in the research effort.

Yuki Nyu: Organized the message board where various decision making by a group of 10 to 30 people were made.Significant experimental results have been obtained from her organizational efforts.

Yoshiharu Maeno, Mr: Developed and implemented the new method human-interactive annealing.Kataichi Ito, Mr: Implemented the basic tool for the experiments of data crystallization

(6) Publications: List peer-reviewed publications submitted and/or accepted during the contract period.

Yoshiharu Maeno and Yukio Ohsawa, Human-Computer Interactive Annealing for Discovering Invisible DarkEvents, submitted to IEEE Transaction on Humatronics (Under review 2006)

Yoshiharu Maeno and Yukio Ohsawa, Understanding of dark events for harnessing risk, Chance

Discovery for Real World Decision Making, Chapter 22, Springer Verlag (2006)Kenichi Horie, Yukio Ohsawa, Product Designed on Scenario Maps Using Pictorial KeyGraph, WSEAS

Transaction on Information Science and Application, Vol.3 No.7, pp.1324-1331 (2006)

Tsuneki Sakakibara, Yukio Ohsawa, Gradual-Increase Extraction of Target Baskets as Preprocess for

Visualizing Simplified Scenario Maps by KeyGraph, Journal of Soft Computing (2006) To Appear

Naohiro Matsumura, Yukio Ohsawaa, Mitsuru Ishizuka, Combination Retrieval for Creating Knowledge from

Sparse Document-Collection, Journal of Knowledge Based Systems, Vol.18, No.7, pp.327 -- 333

(Elsevier, 2006)


5/48

Yukio Ohsawa, Scenario Understanding of Hepatitis Progress and Recovery by Annnotation-based Integration

of Data based Scenario Maps, GESTS International Trans. Computer Science and Engineering Vol.22,

N0.1., pp.65-76 (2005)

Yukio Ohsawa, Data Crystallization: Chance Discovery Extended for Dealing with Unobservable Events,New Mathematics and Natural Computation Vol.1, No.3, pp.373 - 392 (2005)

Renate Fruchter, Yukio Ohsawa, and Naohiro Matsumura, Knowledge reuse through chance discovery from anenterprise design-build enterprise data store, New Mathematics and Natural Computation Vol.1 No.3,

pp.393-406 (2005)

Noriyuki Kushiro, and Yukio Ohsawa, a A scenario acquisition method with multi-dimensional hearing and

hierarchical accommodation process, New Mathematics and Natural Computation Vol.2, No.1, pp.101-

113 (2006)

Xavier Llor, a David E. Goldberg, Yukio Ohsawa, et al, Innovation and Creativity support via ChanceDiscovery, Genetic Algorithms, New Mathematics and Natural Computation, Vol.2, No.1, pp.85-100

(2006)Yukio Ohsawa, Naohiro Matsumura, Naoaki Okazaki Understanding Scenarios of Individual Patients of

Hepatitis in Double Helical Process Involving KeyGraph and DSV, The Fourth IEEE International

Workshop on Soft Computing as Transdisciplinary Science and Technology (WSTST05), Muroran,pp.456- 469 (2005)

Tsuneki Sakakibara, Yukio Ohsawa Knowledge Discovery Method by Gradual Increase of Target Baskets

from Sparse Dataset The Fourth IEEE International Workshop on Soft Computing as Transdisciplinary

Science and Technology (WSTST05), Muroran, pp.480- 489 (2005)

Yuichi Washida, Hiroshi Tamura, Yukio Ohsawa Examining Small World Problem Using KeyGraph The

Fourth IEEE International Workshop on Soft Computing as Transdisciplinary Science and Technology

(WSTST05), Muroran, pp.490- 500 (2005)

(7) Interactions: Please list:

(a) Participation/presentations at meetings, conferences, seminars, etc.

Yukio Ohsawa: "Data Crystallization: A Project Beyond Chance Discovery for Discovering Unobservable

Events," Invited Talk in IEEE International Conference on Granular Computing, Beijin (CDROM, 2005)Yukio Ohsawa: Plenary Lecture "Chance Discovery: Data-based Decision for Design and Business"

International Workshop on Chance Discovery,Aletheia University, Taipei (2005)Yukio Ohsawa: "Data Crystallization: A Project Beyond Chance Discovery for Discovering Unobservable

Events" IEEE International Conference on Granular Computing, Beijin (2005)

Yuko Ohsawa: Designing Systems for Chance Discovery, The Fourth IEEE International Workshop on Soft

Computing as Transdisciplinary Science and Technology, Plenary Lecture (2005)

Yukio Ohsawa, Takaichi Itoh, Data Crystallizer: Tool for Discovering Unobservable Events, 1st Annual

Workshop on Rough Sets and Chance Discovery (RSCD) in conjunction with 8th Joint Conference on

Information Sciences (JCIS 2005), Salt Lake City (2005)

Kazuhisa INABA and Yukio OHSAWA, Study on a Method for Supporting Scenario Extraction from Time

Series Information, 1st Annual Workshop on Rough Sets and Chance Discovery (RSCD) in conjunctionwith 8th Joint Conference on Information Sciences (JCIS 2005), Salt Lake City (2005)

Kenichi HORIE and Yukio OHSAWA, Extracting High Quality Scenario for Consensus On NewSpecifications of Equipment, 1st Annual Workshop on Rough Sets and Chance Discovery (RSCD) in

conjunction with 8th Joint Conference on Information Sciences (JCIS 2005), Salt Lake City (2005)

Yukio Ohsawa, Human-based Annotation of Data-based Scenario Flow on Scenario Map for Understanding

Hepatitis Scenarios, Proc. KES Conference (2005)

Noriyuki Kushiro and Yukio Ohsawa, A Scenario Elicitation Method in Cooperation with Requirements

Engineering and Chance Discovery, Proc. KES Conference (2005)

Calkin A.S. Montero, Yukio Ohsawa, Kenji Araki Modelling the Discovery of Critical Utterances, Proc. KES

Conference (2005)


6/48

Ken-ichi Horie, Yukio Ohsawa, Extracting High Quality Scenario for Consensus on Specifications of New

Products, Proc. KES Conference (2005)

(b) Describe cases where knowledge resulting from your effort is used, or will be used, in a technologyapplication. Not all research projects will have such cases, but please list any that have occurred.

- Visualizing the data of patent lists of a company, with our method of data crystallization, enabled to see new

technologies not yet existing in the world.

(8) New:

(a) List discoveries, inventions, or patent disclosures. (If none, report None.).

- The basic method of data crystallization, enabling to realize hidden leaders and hidden demands in the

market.

- The advanced method of data crystallization, which we call human-interactive annealing.

Patent disclosures: None

(b) Complete the attached DD Form 882, Report of Inventions and Subcontractors.

(9) Honors/Awards: List honors and awards received during the contract period, or emanating from the

AOARD-supported research project.

- Young scientist award, from the Japanese Ministry of Education, Culture, Sports, Science and

Technology (May 2005)

(10) Archival Documentation: This section should include a description of your work at a level of technical

detail that you think to be appropriate. Submission of reprints/preprints often satisfies this requirement. If

you have questions on how to prepare this section, please discuss this matter with your AOARD programmanager.

Attached (the copies of articles below)

Yoshiharu Maeno and Yukio Ohsawa, Understanding of dark events for harnessing risk, Chance

Discovery for Real World Decision Making, Chapter 22m Springer Verlag (2006)

Yukio Ohsawa, Data Crystallization: Chance Discovery Extended for Dealing with Unobservable Events,

New Mathematics and Natural Computation Vol.1, No.3, pp.373 - 392 (2005)


7/48

S e p t e m b e r 1 , 2 0 0 9 : 1 9 W S P C / I N S T R U C T I O N F I L E O h s a w a

N e w M a t h e m a t i c s a n d N a t u r a l C o m p u t a t i o n

c

W o r l d S c i e n t i c P u b l i s h i n g C o m p a n y

D a t a C r y s t a l l i z a t i o n : C h a n c e D i s c o v e r y E x t e n d e d f o r D e a l i n g w i t h

U n o b s e r v a b l e E v e n t s

3

Y u k i o O h s a w a

y

S c h o o l o f E n g i n e e r i n g , T h e U n i v e r s i t y o f T o k y o , 7 - 3 - 1 H o n g o , B u n k y o - k u , 1 1 3 - 8 5 6 3 , J a p a n

y . o h s a w a @ g m a i l . c o m

R e c e i v e d 1 7 J u n e 2 0 0 5

T h i s p a p e r i n t r o d u c e s t h e c o n c e p t o f C h a n c e D i s c o v e r y , i . e . , d i s c o v e r y o f a n e v e n t s i g -

n i c a n t f o r d e c i s i o n m a k i n g . T h e n , t h i s p a p e r a l s o p r e s e n t s a c u r r e n t r e s e a r c h p r o j e c t

o n D a t a C r y s t a l l i z a t i o n , w h i c h i s a n e x t e n s i o n o f C h a n c e D i s c o v e r y . T h e n e e d f o r D a t a

C r y s t a l l i z a t i o n i s t h a t o n l y t h e o b s e r v a b l e p a r t o f t h e r e a l w o r l d c a n b e s t o r e d i n d a t a .

F o r s u c h s c a t t e r e d , i . e . , i n c o m p l e t e a n d i l l - s t r u c t u r e d d a t a , d a t a c r y s t a l l i z i n g a i m s a t p r e -

s e n t i n g t h e h i d d e n s t r u c t u r e a m o n g e v e n t s i n c l u d i n g u n o b s e r v a b l e o n e s . T h i s i s r e a l i z e d

w i t h a t o o l w h i c h i n s e r t s d u m m y i t e m s , c o r r e s p o n d i n g t o u n o b s e r v a b l e b u t s i g n i c a n t

e v e n t s , t o t h e g i v e n d a t a o n p a s t e v e n t s . T h e e x i s t e n c e o f t h e s e u n o b s e r v a b l e e v e n t s a n d

t h e i r r e l a t i o n s w i t h o t h e r e v e n t s a r e v i s u a l i z e d w i t h K e y G r a p h , s h o w i n g e v e n t s b y n o d e s

a n d t h e i r r e l a t i o n s b y l i n k s , o n t h e d a t a w i t h i n s e r t e d d u m m y i t e m s . T h i s v i s u a l i z a t i o n

i s i t e r a t e d w i t h g r a d u a l l y i n c r e a s i n g t h e n u m b e r o f l i n k s i n t h e g r a p h . T h i s p r o c e s s i s

s i m i l a r t o t h e c r y s t a l l i z a t i o n o f s n o w w i t h g r a d u a l d e c r e a s e i n t h e a i r t e m p e r a t u r e . F o r

t u n i n g t h e g r a n u l a r i t y l e v e l o f s t r u c t u r e t o b e v i s u a l i z e d , t h i s t o o l i s i n t e g r a t e d w i t h

h u m a n ' s p r o c e s s o f c h a n c e d i s c o v e r y . T h i s b a s i c m e t h o d i s e x p e c t e d t o b e a p p l i c a b l e f o r

v a r i o u s r e a l w o r l d d o m a i n s w h e r e c h a n c e - d i s c o v e r y m e t h o d s h a v e b e e n a p p l i e d .

K e y w o r d s : U n o b s e r v a b l e E v e n t s ; C h a n c e D i s c o v e r y ; D a t a C r y s t a l l i z a t i o n

1 . I n t r o d u c t i o n

I n t h i s s t u d y , m y r e s e a r c h t e a m i s r e v e a l i n g e v e n t s t h a t a r e p o t e n t i a l l y i m p o r t a n t

b u t h a v e n e v e r b e e n o b s e r v e d . B e c a u s e t h e y a r e n o t i n c l u d e d i n t h e d a t a , e x i s t i n g

m i n i n g m e t h o d s h a r d l y h e l p i n i d e n t i f y i n g s u c h e v e n t s . D a t a c r y s t a l l i z a t i o n i s t h e

c h a l l e n g e t o t h i s d i c u l t p r o b l e m . I t f o r m s a n e x t e n s i o n o f w h a t w e h a v e b e e n

c a l l i n g C h a n c e D i s c o v e r y s i n c e 2 0 0 0

1 2 3

C h a n c e d i s c o v e r y m e a n s t h e d i s c o v e r y o f a c h a n c e , w h i c h i s d e n e d a s a n e v e n t

s i g n i c a n t f o r d e c i s i o n m a k i n g . T h i s h a s b e e n a r e a l c h a l l e n g e t o g o b e y o n d t h e

m e t h o d o l o g y o f d a t a m i n i n g , i n t h a t t h e n e w g o a l i s t h e u n d e r s t a n d i n g o f t h e

3

T h i s w o r k w a s s u p p o r t e d i n p a r t b y t h e U . S . G o v e r n m e n t . M r . T a k a i c h i I t o , K e i o U n i v e r s i t y ,

c o n t r i b u t e d t o t h i s s t u d y a s t h e s o f t w a r e d e v e l o p e r o f d a t a c r y s t a l l i z a t i o n .

y

S c h o o l o f E n g i n e e r i n g , T h e U n i v e r s i t y o f T o k y o , 7 - 3 - 1 H o n g o , B u n k y o - k u , T o k y o 1 1 3 - 8 6 5 3 J a p a n

( e - m a i l : y . o h s a w a @ g m a i l . c o m ) .

1


8/48


2 Y u k i o O h s a w a

m e a n i n g o f r a r e e v e n t s f o r m a k i n g d e c i s i o n s , r a t h e r t h a n l e a r n i n g r u l e s f o r p r e -

d i c t i n g t h e s e r a r e e v e n t s

6 7

. F o r e x a m p l e , d e v e l o p e r s o f c e l l u l a r p h o n e a r e s e e k i n g

c o m m e n t s f r o m u s e r s . S o m e c o m m e n t s s i g n i c a n t l y a e c t t h e d e c i s i o n o f a d e v e l -

o p e r t o r e d e s i g n c e l l u l a r p h o n e s , s o t h e y c a n b e r e g a r d e d a s \ c h a n c e s . " G i v e n t h e s e

c o m m e n t s , d a t a / t e x t m i n i n g t o o l s m a y b e a b l e t o s h o w t h e r e l a t i o n s b e t w e e n c o m -

m e n t s , t h e s i m i l a r i t i e s o f u s e r s , e t c . O n t h e o t h e r h a n d , m e t h o d s o f c h a n c e d i s c o v e r y

a i d h u m a n - c o m p u t e r i n t e r a c t i o n s t o p o t e n t i a l l y a c h i e v e t h e d e t e c t i o n o f r a r e b u t i n -

u e n t i a l e v e n t s / w o r d s / i t e m s / p e o p l e

8 9 1 0

. I n o r d e r t o r e a l i z e C h a n c e D i s c o v e r y , w e

d e v e l o p e d t o o l s o f d a t a - v i s u a l i z a t i o n

1 1 1 2

, t o b e c o u p l e d w i t h h u m a n ' s p e r c e p t i o n

o f c h a n c e s

1 3

. I n t h e n e x t s e c t i o n , w e w i l l r e v i e w p r e v i o u s a p p r o a c h e s t o C h a n c e

D i s c o v e r y .

2 . T h e P r o b l e m o f C h a n c e D i s c o v e r y

L e t u s d e n e a s c e n a r i o a s a s e q u e n c e o f e v e n t s a n d a c t i o n s i n a c e r t a i n c o n t e x t . F o r

e x a m p l e , s u p p o s e a c u s t o m e r o f a d r u g s t o r e b u y s a n u m b e r o f i t e m s i n s e r i e s , a f e w

i t e m s p e r m o n t h . H e h a s a n u r g e t o d o s o b e c a u s e h e h a s a c e r t a i n p e r s i s t e n t d i s e a s e .

I n t h i s c a s e , f u l l l i n g a r e m e d y o f t h e d i s e a s e s u g g e s t e d b y h i s d o c t o r i s t h e p u r p o s e

c o v e r i n g t h e e n t i r e e v e n t - s e q u e n c e , w h e r e a n e v e n t i s t h e p a t i e n t ' s p u r c h a s e o f a

d r u g . H e r e , t h e p u r p o s e t o f u l l l t h e r e m e d y i s t h e c o n t e x t c o v e r i n g t h e s e q u e n c e .

T h e n , t h i s p a t i e n t m a y l e a r n s a b o u t a n e w d r u g , a n d s t a r t s t o t a k e i t f o r c h a n g i n g

t h e s c e n a r i o t o a r a d i c a l c u r e . A f t e r a m o n t h , h i s d o c t o r g e t s u p s e t h e a r i n g t h i s

c h a n g e i n t h e t r e a t m e n t d u e t o t h e p a t i e n t ' s i g n o r a n c e r e g a r d i n g t h e r i s k o f t h e

n e w d r u g . H e r e , t h e d o c t o r n o t i c e d t h e r i s k y s c e n a r i o i n t h e c o n t e x t o f s i d e e e c t s .

T h e d o c t o r u r g e n t l y i n t r o d u c e s s u r g i c a l o p e r a t i o n , a p o w e r f u l m e t h o d t o o v e r c o m e

t h e s i d e e e c t s a n d c h a n g e i n t o t h e t h i r d s c e n a r i o i n t h e c o n t e x t o f r e c o v e r y .

I n t h i s e x a m p l e , w e n d t w o \ c h a n c e s " i n t h e t h r e e s c e n a r i o s . T h e r s t c h a n c e i s

t h e i n f o r m a t i o n a b o u t t h e n e w d r u g w h i c h c h a n g e s f r o m t h e r s t r e m e d y s c e n a r i o

t o t h e s e c o n d s c e n a r i o , i . e . , t h e r i s k y o n e . T h e n t h e d o c t o r ' s s u r p r i s e b e c a m e t h e

s e c o n d c h a n c e t o t u r n t o t h e t h i r d s c e n a r i o . A c c o r d i n g t o t h e d e n i t i o n o f \ c h a n c e "

b y O h s a w a

1

, i . e . , a n e v e n t o r a s i t u a t i o n s i g n i c a n t f o r d e c i s i o n m a k i n g , a c h a n c e

o c c u r s a t t h e c r o s s p o i n t o f m u l t i p l e s c e n a r i o s a s i n t h e e x a m p l e a b o v e , b e c a u s e

a d e c i s i o n i s t o s e l e c t o n e s c e n a r i o i n t h e f u t u r e . B a s e d o n t h i s i d e a , m e t h o d s o f

C h a n c e D i s c o v e r y m a y c o n t r i b u t e s i g n i c a n t l y t o s c i e n c e s a n d b u s i n e s s d o m a i n s

3

H e r e , l e t u s s t a n d o n t h e p o s i t i o n o f a p h y s i c i a n l o o k i n g a t t h e t i m e s e r i e s o f

s y m p t o m s d u r i n g t h e p r o g r e s s o f a n i n d i v i d u a l p a t i e n t ' s d i s e a s e . T h e p h y s i c i a n

s h o u l d t a k e a p p r o p r i a t e a c t i o n s f o r c u r i n g t h i s p a t i e n t , a t a p p r o p r i a t e t i m e s .

S c e n a r i o 1 = e v e n t 1 ! e v e n t 2 ! e v e n t 3 ( t h e p r o g r e s s o f t h e d i s e a s e )

S c e n a r i o 2 = e v e n t 4 ! e v e n t ! e v e n t 6 ( t h e e f f e c t o f t h e n e w d r u g ) ( 2 . 1 )

E a c h e v e n t - s e q u e n c e i n E q . ( 2 . 1 ) i s a s c e n a r i o a s f a r a s i t i s c o v e r e d b y s o m e

c o h e r e n t c o n t e x t . F o r e x a m p l e , S c e n a r i o 1 i s i n t h e c o n t e x t o f d i s e a s e p r o g r e s s i o n

w i t h o u t t r e a t m e n t , a n d S c e n a r i o 2 i s a s c e n a r i o i n t h e c o n t e x t o f t a k i n g a n e w d r u g


9/48


D a t a C r y s t a l l i z a t i o n B e y o n d C h a n c e D i s c o v e r y 3

w i t h a s i d e e e c t . S u p p o s e t h e r e i s a n o t h e r e v e n t 9 , m e a n i n g t h e a p p e a r a n c e o f t h e

n e w d r u g , s h o r t l y a f t e r e v e n t 2 . T h e p a t i e n t t o o k t h i s a s a g o o d c h a n c e , b y j u s t

l o o k i n g a t t h e l o c a l r e l a t i o n a m o n g e v e n t 2 , e v e n t 9 , a n d e v e n t 4 . F o r t h i s p a t i e n t ' s

p e r c e p t i o n , t h e a p p e a r a n c e o f e v e n t 9 j u s t a f t e r e v e n t 2 b e c a m e e s s e n t i a l f o r m a k i n g

a d e c i s i o n , a n d l o o k e d l i k e a s i g n i c a n t c h a n c e . H o w e v e r , t h e d o c t o r l o o k e d a t t h e

o v e r a l l r e l a t i o n s a m o n g a l l e v e n t s i n t h e m a p i n F i g . 1 , a n d n o t i c e d t h e p a t i e n t i s

g o i n g i n a w r o n g d i r e c t i o n . T h a n k s t o h i s a w a r e n e s s o f a s i d e e e c t ( e v e n t ) o f t h e

n e w d r u g , h e d e c i d e s t o p e r f o r m a s u r g i c a l o p e r a t i o n .

D e t e c t i n g a n e v e n t a t a c r o s s p o i n t b e t w e e n m u l t i p l e s c e n a r i o s , s u c h a s e v e n t

2 , e v e n t 9 , a n d e v e n t a b o v e , a n d s e l e c t i n g t h e s c e n a r i o t h a t i n c l u d e s s u c h a c r o s s

p o i n t i s t h e e s s e n c e o f C h a n c e D i s c o v e r y . I n g e n e r a l , t h e m e a n i n g o f a s c e n a r i o

w i t h a n e x p l a n a t o r y c o n t e x t i s e a s i e r t o u n d e r s t a n d t h a n a n e v e n t s h o w n a l o n e .

F r o m F i g . 1 , w e c a n u n d e r s t a n d t h e t h r e e b a s i c s c e n a r i o s , a n d t h e n o v e l s c e n a r i o

e m e r g i n g f r o m c o n n e c t i n g t h e b a s i c s c e n a r i o s v i a c h a n c e e v e n t s . H o w e v e r , e v e n t 2 ,

e v e n t 9 , a n d e v e n t a s s h o w n i n F i g . 1 , a r e h a r d e r t o u n d e r s t a n d i f t h e y a r e s h o w n

i n d e p e n d e n t l y o f o t h e r e v e n t s . W i t h o u t t h i s u n d e r s t a n d i n g , i t w o u l d b e d i c u l t

t o o b t a i n t h e p a t i e n t ' s c o n s e n s u s o n i n t r o d u c i n g t h e s u r g i c a l o p e r a t i o n , b e c a u s e a

r a r e e v e n t s u c h a s e v e n t 9 m a k e s t h e s i t u a t i o n h a r d e r t o a c c e p t , a n d b e c a u s e t h i s

s u r g i c a l o p e r a t i o n i t s e l f i s r a r e f o r o r d i n a r y p a t i e n t s .

F o r r e a l i z i n g s u c h a n u n d e r s t a n d i n g , v i s u a l i z i n g t h e s c e n a r i o m a p i . e . a t w o -

d i m e n s i o n a l g r a p h o n w h i c h u s e r c a n n d a m e a n i n g f u l s c e n a r i o b y n d i n g a c o n t e x t

c o v e r i n g a c o n n e c t e d s e q u e n c e o f e v e n t s , i s u s e f u l . F o r e x a m p l e , o n t h e s c e n a r i o m a p

i n F i g . 1 , u s e r c a n n d t h e c o n n e c t e d s c e n a r i o b e g i n n i n g f r o m S c e n a r i o 1 , t o m o v e o n

v i a S c e n a r i o 2 , a n d , n a l l y , t o r e a c h S c e n a r i o 3 . H e r e , w e c a n r e g a r d e a c h f a m i l i a r

s c e n a r i o , s u c h a s S c e n a r i o 1 o r S c e n a r i o 2 , a s a n i s l a n d . A n d , l e t u s r e g a r d a p a t h

o f l i n k s b e t w e e n i s l a n d s a s a b r i d g e . I n C h a n c e D i s c o v e r y , t h e p r o b l e m t h e n i s t o

h a v e t h e u s e r o b t a i n i n g b r i d g e s b e t w e e n i s l a n d s , i n o r d e r t o e x p l a i n t h e m e a n i n g

o f c o n n e c t i o n s b e t w e e n i s l a n d s b y m e a n s o f b r i d g e s , a s a s c e n a r i o w h i c h c a n b e

e x p r e s s e d i n a l a n g u a g e t h a t i s u n d e r s t a n d a b l e f o r t h e u s e r h i m s e l f / h e r s e l f .

3 . T h e H u m a n - M a c h i n e I n t e r a c t i o n i n C h a n c e D i s c o v e r y

I n t h e p r e v a l e n t t e r m \ s c e n a r i o d e v e l o p m e n t , " a s c e n a r i o m a y s o u n d l i k e s o m e t h i n g

t o b e \ d e v e l o p e d " b y h u m a n s w h o c o n s c i o u s l y c o n t r o l t h e p r o c e s s b y p l a n n i n g

a c t i o n s . H o w e v e r , v a l u a b l e s c e n a r i o s m a y o f t e n \ e m e r g e " u n c o n s c i o u s l y f r o m c o m -

m u n i c a t i o n s o f h u m a n s . F o r e x a m p l e , a s c e n a r i o w o r k s h o p d e v e l o p e d b y t h e D a n i s h

B o a r d o f T e c h n o l o g y ( 2 0 0 3 ) s t a r t s f r o m s c e n a r i o s o f t h e f u t u r e s o c i e t y t h a t a r e p r e -

s e t b y w r i t e r s , t h e n e x p e r t s i n t h e d o m a i n c o r r e s p o n d i n g t o t h e p r e s e t s c e n a r i o s

d i s c u s s s c e n a r i o s f o r a c h i e v i n g f u r t h e r i m p r o v e m e n t s . T h e d i s c u s s a n t s w r i t e d o w n

t h e i r o p i n i o n s d u r i n g t h e w o r k s h o p , b u t i t i s r a r e t h a t t h e y n o t i c e a l l t h e r e a s o n s w h y

t h o s e o p i n i o n s c a m e o u t a n d w h y t h e r e v i s e d s c e n a r i o s h a v e b e e n n a l l y o b t a i n e d .

T h i s p r o c e s s o f a s c e n a r i o w o r k s h o p c a n b e c o m p a r e d w i t h t h e K J ( K a w a k i t a

J i r o ) m e t h o d . I n t h e K J m e t h o d , p a r t i c i p a n t s w r i t e d o w n t h e i r i n i t i a l i d e a s o n


10/48



F i g . 1 . A c h a n c e t h a t e x i s t s a t t h e c r o s s p o i n t o f s c e n a r i o s . T h e s c e n a r i o i n t h e t h i c k a r r o w s

e m e r g e d f r o m S c e n a r i o 1 a n d S c e n a r i o 2 .

K J c a r d s a n d h e n c e a r r a n g e t h e c a r d s i n a 2 D - s p a c e , i n c o - w o r k i n g f o r n d i n g a

g o o d p l a n o f a c t i o n s . H e r e , t h e i d e a o n e a c h c a r d r e e c t s t h e f u t u r e s c e n a r i o i n a

p a r t i c i p a n t ' s m i n d . T h e n e w c o m b i n a t i o n o f p r o p o s e d s c e n a r i o s , g e n e r a t e d d u r i n g

t h e a r r a n g e m e n t a n d t h e r e a r r a n g e m e n t s o f K J c a r d s , h e l p s t h e e m e r g e n c e o f n e w

v a l u a b l e s c e n a r i o s . I n s o m e d e s i g n p r o c e s s e s , o n t h e o t h e r h a n d , i t h a s b e e n p o i n t e d

o u t t h a t a m b i g u o u s i n f o r m a t i o n c a n t r i g g e r c r e a t i o n s

4

. T h e c o m m o n p o i n t a m o n g

t h e s c e n a r i o \ w o r k s h o p " , t h e \ c o m b i n a t i o n " o f i d e a s i n t h e K J m e t h o d , a n d t h e

\ a m b i g u i t y " o f t h e i n f o r m a t i o n t o a d e s i g n e r i s t h a t s c e n a r i o s p r e s e n t e d f r o m t h e

v i e w p o i n t o f e a c h p a r t i c i p a n t ' s e n v i r o n m e n t , a r e b r i d g e d v i a a m b i g u o u s p i e c e s o f

i n f o r m a t i o n a b o u t d i e r e n t m e n t a l w o r l d s , w h i c h t h e p a r t i c i p a n t s a t t e n d . F r o m

t h e s e b r i d g e s , e a c h p a r t i c i p a n t i n d e e d r e c o g n i z e s s i t u a t i o n s o r e v e n t s w h i c h m a y

w o r k a s \ c h a n c e s " i . e . , c r o s s - o v e r p o i n t s f o r f u s i n g o t h e r s ' s c e n a r i o s w i t h o n e ' s o w n .

T h i s c a n b e e x t e n d e d t o o t h e r d o m a i n s t h a n d e s i g n i n g . I n t h e e x a m p l e o f F i g . 1 ,

t h e h o p e f u l S c e n a r i o 3 a f t e r e v e n t m a y b e p r o p o s e d b y t h e d o c t o r , a n d c o n n e c t e d

w i t h S c e n a r i o 2 c h o s e n b y t h e p a t i e n t b e f o r e e v e n t . H e r e , e v e n t p l a y e d t h e r o l e

o f c r o s s - o v e r p o i n t o f t h e t w o s c e n a r i o s , o r t h e s t a r t i n g p o i n t o f t h e t h i c k a r r o w

b r i d g e .

I n t h e s t u d i e s o f C h a n c e D i s c o v e r y , t h e d i s c o v e r y p r o c e s s h a s b e e n s u p p o s e d

b y O h s a w a t o f o l l o w t h e D o u b l e H e l i x ( D H ) m o d e l

1 3

a s s h o w n i n F i g . 2 ( D a t a

C r y s t a l l i z a t i o n i n F i g . 2 i s t o b e e x p l a i n e d i n l a t e r s e c t i o n s ) . T h e D H p r o c e s s s t a r t s

f r o m t h e i n i t i a l s t a t e o f t h e u s e r ' s m i n d t h a t i s c o n c e r n e d w i t h c a t c h i n g a n e w


11/48



c h a n c e . T h i s c o n c e r n i s r e e c t e d t o a c q u i r i n g e x t e r n a l d a t a t o b e a n a l y z e d b y a

d a t a - v i s u a l i z i n g t o o l s u c h a s K e y G r a p h ( t o a p p e a r i n l a t e r s e c t i o n s ) , w h i c h h a s

b e e n s p e c i c a l l y d e s i g n e d f o r C h a n c e D i s c o v e r y . T h e v i s u a l i z a t i o n t o o l m a y d e p i c t

e a c h i t e m i n t h e d a t a a s a n o d e , a n d t h e c o - o c c u r r e n c e b e t w e e n i t e m s m a y b e s h o w n

a s l i n k s a m o n g n o d e s . S u c h a d i a g r a m h a s b e e n r e g a r d e d a s a s c e n a r i o m a p l i k e

F i g . 1 .

F i g . 2 . D a t a c r y s t a l l i z a t i o n o n t h e n d o u b l e h e l i x p r o c e s s .

L o o k i n g a t t h e s c e n a r i o m a p o b t a i n e d , p o s s i b l e s c e n a r i o s a n d t h e i r m e a n i n g s

e m e r g e i n e a c h u s e r ' s m i n d . T h e n , u s e r s p a r t i c i p a t e i n a c o - w o r k i n g g r o u p f o r C h a n c e

D i s c o v e r y , s h a r i n g t h e s a m e s c e n a r i o m a p . H e r e , t h e y p r e s e n t t h e s c e n a r i o s t h e y n d

f r o m t h e m a p . A s a r e s u l t , t h e c o m p u t e r a c q u i r e s i n t e r n a l d a t a i . e . t h e t e x t d a t a

r e c o r d i n g t h e t h o u g h t s a n d o p i n i o n s p r e s e n t e d i n t h e d i s c u s s i o n . T h e v i s u a l i z a t i o n

t o o l i s u s e d n o w a g a i n : W o r d s c o r r e s p o n d i n g t o c o n t e x t u a l b r i d g e s a r e v i s u a l i z e d ,

c o n n e c t e d w i t h p r e v a l e n t d a i l y - l i f e c o n t e x t s o f p a r t i c i p a n t s . B y t h i s t i m e , t h e p a r -

t i c i p a n t s d i s c o v e r c h a n c e s o n t h e b r i d g e s . B a s e d o n t h e s e c h a n c e s , t h e u s e r s c a n

m a k e a n e w d e c i s i o n i n t h e r e a l w o r l d . F i n a l l y , t h e u s e r s p e r f o r m a r e a l a c t i o n o n

w h i c h t h e y o b t a i n c o n c e r n s w i t h n e w c h a n c e s , a n d t h e h e l i c a l p r o c e s s r e t u r n s t o

t h e i n i t i a l s t e p o f t h e n e x t c y c l e .

I n t h e c a s e o f m a r k e t i n g , p a r t i c i p a n t s o f a b u s i n e s s m e e t i n g r a n o n t h e D H


12/48



p r o c e s s w i t h s h a r i n g t h e r e s u l t o f K e y G r a p h . T h e y l o o k e d a t t h e m a p o f t h e i r m a r -

k e t u s i n g K e y G r a p h , w h e r e n o d e s c o r r e s p o n d t o p r o d u c t s a n d l i n k s c o r r e s p o n d i n g

t o c o - o c c u r r e n c e s b e t w e e n p r o d u c t s i n t h e c u s t o m e r ' s b a s k e t d a t a . O n t h i s m a p ,

p a r t i c i p a n t s ( m a r k e t r e s e a r c h e r s ) d i s c u s s e d w i t h e x c h a n g i n g s c e n a r i o s o f c u s t o m e r s

l i v i n g o n v a r i o u s p r o d u c t - s e g m e n t s c o r r e s p o n d i n g t o l o c a l i s l a n d s i n t h e m a p . A s a

r e s u l t , t h e y f o u n d n e w s c e n a r i o s o f l i v i n g c u s t o m e r s w h o m a y b u y p r o d u c t s i n a l l

o v e r t h e w i d e m a r k e t . I n c o n t r a s t , p r e v i o u s m e t h o d s o f d a t a - b a s e d m a r k e t i n g c o u l d

i d e n t i f y f o c u s e d s e g m e n t s o f p r o d u c t s a n d t h e s c e n a r i o s i n e a c h l o c a l s e g m e n t . T h i s

r e a l i z e d t h e h i t s o f n e w p r o d u c t s a p p e a r i n g i n K e y G r a p h a t b r i d g e s b e t w e e n i s l a n d s .

T h u s , t h e p a r t i c i p a n t s o f t h e D H p r o c e s s r e a l l y d i s c o v e r e d r e m a r k a b l e c h a n c e s , a n d

m a d e r e a l b u s i n e s s p r o t s

8

4 . D a t a C r y s t a l l i z a t i o n : A N e w C h a l l e n g e

T h e c o m p l e x i t y o f t h e r e a l w o r l d w a s s o m e t i m e s b e y o n d t h e r e a c h o f p r e v i o u s m e t h -

o d s f o r C h a n c e D i s c o v e r y : A f e w n e r d u s e r s o f c e l l u l a r p h o n e s , w h o d o n o t s e n d o u t

c o m m e n t s f r e q u e n t l y a b o u t t h e i r w a y o f u s i n g c e l l u l a r , a r e l i k e l y t o c r e a t e a n e w

f a s h i o n c a u s i n g s t r o n g i n u e n c e s o n o t h e r u s e r s . T h e d e v e l o p e r ' s q u e s t i o n i s \ w h e r e

i s t h e i n n o v a t i v e u s e r ? " I f a n s w e r s t o t h e s e q u e s t i o n s a r e a v a i l a b l e , t h e d e v e l o p e r

c a n c o n t i n u e t o o b s e r v e t h e b e h a v i o r s o f t h e i n n o v a t i v e u s e r , a n d m a y b e a b l e t o

c a t c h t h e s i g n s o f n e w t r e n d s . T h i s c a n b e a s i g n i c a n t c h a n c e i n b u s i n e s s , t h a t

m a y a e c t h i s d e c i s i o n .

I t i s m e a n i n g l e s s t o a s k h u n d r e d s o f m o n i t o r s \ w h o g a v e y o u t h e i d e a t o u s e

c e l l u l a r p h o n e s i n t h i s w a y ? " b e c a u s e u s e r s s e l d o m s e e i n n o v a t i v e u s e r s , b u t o n l y

s e e o t h e r u s e r s ' a c c e s s o r i e s o f c e l l u l a r w h i c h a r e t h e i n d i r e c t e e c t s o f t h e i n n o v a t i o n .

A s a r e s u l t , n e i t h e r c o m m e n t s n o r n a m e s o f i n n o v a t o r s c a n b e i n c l u d e d i n t h e d a t a

o n u s e r ' s c o m m e n t s . H e r e a r o s e t h e p r o b l e m o f D a t a C r y s t a l l i z a t i o n .

D a t a C r y s t a l l i z a t i o n , o u r n e w p r o j e c t t h a t e x t e n d s C h a n c e D i s c o v e r y , i s d e d i -

c a t e d t o e x p e r t s w o r k i n g i n r e a l d o m a i n s w h e r e d i s c o v e r i e s o f u n o b s e r v a b l e e v e n t s

a r e d e s i r e d . F o r e x a m p l e , l e t u s c o n s i d e r i n t e l l i g e n c e a n a l y s i s , w h e r e e x p e r t i n v e s -

t i g a t o r s o f c r i m i n a l - g r o u p b e h a v i o r s a r e e x p l o r i n g l i n k s a m o n g m e m b e r s . T h e t o p

l e a d e r ( s e e t h e d a r k m a n a t t h e t o p o f F i g . 3 ) o f t h e c r i m i n a l o r g a n i z a t i o n m a y

p h o n e a f e w t i m e s t o s u b - l e a d e r s m a n a g i n g l o c a l s e c t i o n s ( M r . A a n d M r . B i n

F i g . 3 ) . F o r r e s p o n d i n g t o t h e s e t o p - l e v e l c o m m a n d s , e a c h l o c a l s e c t i o n h o l d s i t s

i n t e r n a l c o m m u n i c a t i o n , v i a d i e r e n t m e d i a f r o m t h a t t h e t o p l e a d e r u s e d f o r c o n -

t a c t i n g s u b - l e a d e r s . T h e n , t h e s u b - l e a d e r s m a y m e e t t o a c h i e v e c o n s e n s u s b e f o r e

r e s p o n d i n g t o t h e t o p l e a d e r . M e a n w h i l e , t h e l e a d e r d o e s n o t a p p e a r i n a n y m e e t -

i n g s . I n t h i s w a y , s o m e o n e n e v e r o b s e r v e d i n m e e t i n g s o r m a i l i n g l i s t s m a y b e t h e

a c t u a l l e a d e r .

5 . T h e M e t h o d O v e r v i e w o f D a t a C r y s t a l l i z a t i o n

T h e o b j e c t i v e o f D a t a C r y s t a l l i z a t i o n i s t o d e t e c t ( n o t o n l y r a r e b u t ) u n o b s e r v a b l e

s i g n i c a n t e v e n t s . I n t h i s p a p e r , I p r e s e n t a n a p p r o a c h i n t e g r a t i n g t w o n e w m e t h o d s ,


13/48



t o a b r e a k t h r o u g h f r o m t h e c u r r e n t s s t a t e o f a r t i n C h a n c e D i s c o v e r y .

T h e r s t i s a m e t h o d o f v i s u a l i z i n g d a t a b y i n s e r t i n g a r t i c i a l d u m m y i t e m s .

T h e s e d u m m y i t e m s m e a n u n o b s e r v a b l e e v e n t s , o f w h i c h t h e e n t i t i e s a r e t o t a l l y

u n k n o w n . T h e s e c o n d i s t h e h u m a n ' s p r o c e s s o f d i s c o v e r y , w h e r e t h e c h a n c e m a y

n o t b e i n c l u d e d i n t h e d a t a . F o r e x a m p l e , i f t h e l e a d e r o f a c r i m i n a l g r o u p i s u n o b -

s e r v a b l e , t h e i n t e l l i g e n c e a n a l y s t s h o u l d b e c o m e c o n c e r n e d w i t h s o m e o n e c o n t a c t i n g

s u b - l e a d e r s m o d e r a t i n g l o c a l m e e t i n g s ( M r . A a n d M r . B i n F i g . 3 ) . T h e n , t h e a n a -

l y s t m a y m o v e t o t h e s t e p o f o b s e r v i n g t h e l i v i n g e n v i r o n m e n t s o f M r . A a n d M r .

B . I n t h i s w a y , h u m a n ' s i n t e r a c t i o n w i t h t h e r e a l w o r l d s h o u l d b e p o s i t i o n e d i n t h e

p r o c e s s o f d a t a c r y s t a l l i z a t i o n .

B a s i c a l l y , t h e p r e s e n t e d m e t h o d f o l l o w s t h e D o u b l e H e l i x p r o c e s s a s i n F i g . 2 ,

w h i c h h a d b e e n o r i g i n a l l y d e v e l o p e d f o r C h a n c e D i s c o v e r y

1 3

a n d m o d i e d s p e c i f -

i c a l l y f o r D a t a C r y s t a l l i z a t i o n . I t b e g i n s w i t h u s e r ' s i n i t i a l c o n c e r n w i t h o c c u r r i n g

e v e n t s w h i c h m a y b e c h a n c e s . O n t h i s c o n c e r n , h e / s h e c o l l e c t s d a t a f r o m t h e e n v i -

r o n m e n t . T h e d a t a a r e v i s u a l i z e d i n t h e c o m p u t e r - g e n e r a t e d M a p 1 o f F i g . 2 , s h o w i n g

t h e c o m p u t e d r e l a t i o n s b e t w e e n e v e n t s i n t h e r e a l w o r l d , a n d t h e u s e r b e g i n s t o t h i n k

o f p o s s i b l e s c e n a r i o s b y c o n n e c t i n g t h e e v e n t s v i s u a l i z e d . H i s / h e r t h o u g h t h e r e , o r

t h e c o m m u n i c a t i o n o f p e o p l e w o r k i n g t o g e t h e r , a r e s t o r e d i n t e x t . T h i s t e x t m e a n s

s t o r i e s r i s i n g f r o m u s e r ' s r e a l - l i f e e x p e r i e n c e s c o r r e s p o n d i n g t o t h e s c e n a r i o s d r a w n

i n M a p 1 . T h i s t e x t i s t h e n v i s u a l i z e d i n M a p 2 . B y l o o k i n g a t M a p 2 , p o s s i b l e

s c e n a r i o s c o m p o s e d o f a s e q u e n c e o f e v e n t s i n c l u d i n g u n o b s e r v a b l e c h a n c e s b e c o m e

e x t e r n a l i z e d . T h i s l e t s t h e u s e r b e c o m e c o n c e r n e d w i t h a c e r t a i n p a r t o f t h e r e a l

e n v i r o n m e n t , a n d b r i n g s t h e u s e r t o t h e s t a r t o f t h e n e x t c y c l e o f t h e h e l i c a l p r o c e s s .

T h e e e c t o f t h i s p r o c e s s , t o t u n i n g t h e g r a n u l a r i t y o f i n f o r m a t i o n a b o u t c h a n c e s ,

e n a b l e d a p p l i c a t i o n s s u c h a s s e l l i n g n e w p r o d u c t s i n m a r k e t i n g

8

, d e t e c t i n g e a r t h -

q u a k e s i g n s

1 4

, t r e a t m e n t o p p o r t u n i t y o f h e p a t i t i s

9

e t c . F o r D a t a C r y s t a l l i z a t i o n ,

w e e x t e n d t h i s p r o c e s s b y p u t t i n g t h e d u m m y - b a s e d v i s u a l i z a t i o n t o M a p 1 a n d M a p

2 . I n t h i s w a y , w e a i m a t r e s o l v i n g h a r d e r p r o b l e m s t h a n w e c h a l l e n g e d s o f a r : D i s -

c o v e r y o f u n o b s e r v a b l e c r i m i n a l l e a d e r s , r e v e a l i n g l a t e n t i n n o v a t o r s , u n o b s e r v a b l e

s y m p t o m s o f h e p a t i t i s , u n o b s e r v a b l e a c t i v e f a u l t s o f e a r t h q u a k e s , e t c .

6 . K e y G r a p h : T h e B a s i c T o o l f o r V i s u a l i z i n g S c e n a r i o M a p s

K e y G r a p h

1 1 1 2

i s a t o o l w e h a d d e v e l o p e d f o r v i s u a l i z i n g r e l a t i o n s a m o n g d a t a

i t e m s , c o r r e s p o n d i n g t o e v e n t s i n t h e r e a l w o r l d . I f t h e e n v i r o n m e n t h e r e m e a n s t h e

s o c i e t y a t t a c k e d b y t h e t e a m w o r k o f a c r i m i n a l g r o u p , K e y G r a p h s h o w s t h e r e l a t i o n

o f t h e g r o u p ' s m e m b e r s o n t h e c o - e x i s t i n g f r e q u e n c i e s a m o n g m e m b e r s . I n E q . ( 6 . 2 ) ,

l e t d a t a D 1 e x p r e s s a s e t o f m e e t i n g s , i n s e r t i n g a p e r i o d ( \ . " ) a t e a c h e n d o f a

m e e t i n g . H e r e , \ m e m b e r 1 " i n E q . ( 6 . 2 ) c a n b e r e g a r d e d a s a n e v e n t t h a t a m e m b e r

a p p e a r e d i n a m e e t i n g p l a c e . R e g a r d i n g e a c h i t e m i n t h e d a t a a s a n e v e n t r a t h e r

t h a n a n o b j e c t i s m e a n i n g f u l i n i n t e r p r e t i n g K e y G r a p h a s a s c e n a r i o m a p , w h e r e

t h e s e q u e n c e o f e v e n t s s h o u l d b e g r a s p e d f r o m t h e c o n n e c t i o n s b e t w e e n n o d e s .


14/48



D 1 = ( s e t 1 ) m e m b e r 1 m e m b e r 2 m e m b e r 3

( s e t 2 ) m e m b e r 1 m e m b e r 2 m e m b e r 3 m e m b e r 4

( s e t 3 ) m e m b e r 4 m e m b e r m e m b e r 7 m e m b e r 6

( s e t 4 ) m e m b e r m e m b e r 2 m e m b e r 3 m e m b e r 7 m e m b e r 6

( s e t ) m e m b e r 1 m e m b e r 2 m e m b e r 7 m e m b e r 6 m e m b e r 9

( s e t 6 ) m e m b e r m e m b e r 7 m e m b e r 6 m e m b e r 9 ( 6 . 2 )

K e y G r a p h t a k e s t h e f o l l o w i n g s t e p s , a n d i s a p p l i e d t o d a t a i n t h e f o r m o f D 1

C o n s e q u e n t l y , F i g . 4 i s o b t a i n e d .

K e y G r a p h - S t e p 1 : T h e M

1

m o s t f r e q u e n t i t e m s i n t h e d a t a ( e . g . , \ m e m b e r 1 " i n

E q . ( 6 . 2 ) ) a r e d e p i c t e d w i t h b l a c k n o d e s . T h e M

2

m o s t s t r o n g l y c o - o c c u r r i n g

i t e m - p a i r s ( i . e . , t h e p a i r s o f t h e h i g h e s t v a l u e s o f t h e J a c c a r d c o - e c i e n t J

i n E q . ( 6 . 3 ) ) g e t l i n k e d v i a b l a c k l i n e s .

J ( X Y ) = p ( X \ Y ) = p ( X [ Y ) ( 6 . 3 )

H e r e , p ( X \ Y ) m e a n s t h e p r o b a b i l i t y t h a t b o t h i t e m X a n d i t e m Y

a p p e a r i n t h e s a m e l i n e s i n d a t a ( a s i n D 1 i n E q . ( 6 . 2 ) ) . p ( X \ Y ) c a n

b e c o m p u t e d b y d i v i d i n g t h e n u m b e r o f l i n e s i n c l u d i n g b o t h X a n d Y b y

t h e n u m b e r o f a l l l i n e s i n t h e d a t a . S i m i l a r l y p ( X [ Y ) i s d e n e d t o m e a n

t h e p r o b a b i l i t y t h a t e i t h e r i t e m X o r i t e m Y a p p e a r s i n t h e s a m e l i n e s

i n d a t a . F o r e x a m p l e , m e m b e r 1 , m e m b e r 2 , a n d m e m b e r 3 i n E q . ( 6 . 2 ) a r e

c o n n e c t e d w i t h b l a c k l i n e s i n F i g . 4 . E a c h c o n n e c t e d g r a p h h e r e f o r m s o n e

i s l a n d i m p l y i n g a b a s i c c o n t e x t o f t h e b e l o n g i n g m e m b e r s ' l i f e .

K e y G r a p h - S t e p 2 : T h e M

3

i t e m s c o - o c c u r r i n g w i t h i s l a n d s i n t h e m a p m o s t

s t r o n g l y , i . e . , X o f t h e l a r g e s t k e y ( X ) i n E q . ( 6 . 4 ) , a r e o b t a i n e d a s h u b s .

F o r e x a m p l e , m e m b e r 9 i n E q . ( 6 . 2 ) i s o b t a i n e d h e r e a s a h u b .

k e y ( X ) = 1 0 5

Y : e a c h i s l a n d

f 1 0 J ( X Y ) g ( 6 . 4 )

T h a t i s , t h e s t r e n g t h h e r e b e t w e e n i t e m X a n d i s l a n d Y i s c o m p u t e d

a s J a c c a r d c o - e c i e n t , a f t e r c h a n g i n g t h e n a m e o f e a c h i t e m i n a n i s l a n d

i n t o t h e n a m e o f t h e i s l a n d , i n t h e g i v e n d a t a . F o r e x a m p l e , i f m e m b e r 1 i s

i n c l u d e d i n t h e r s t i s l a n d , s o i t i s r e n a m e d i n t o i s l a n d 1 . I f m e m b e r i s i n

t h e s e c o n d i s l a n d , i t i s r e n a m e d i n t o i s l a n d 2 , i n D 1 . T h e n , t h e c o - o c c u r r e n c e

s t r e n g t h b e t w e e n m e m b e r 9 a n d i s l a n d 1 i s c o m p u t e d o n E q . ( 6 . 3 ) , a n d i s

u s e d i n E q . ( 6 . 4 ) . I n t h e o b t a i n e d r e s u l t , a p a t h o f l i n k s c o n n e c t i n g i s l a n d s

v i a h u b s i s c a l l e d a b r i d g e . I f a h u b i s r a r e r t h a n b l a c k n o d e s , i t i s c o l o r e d

i n a d i e r e n t c o l o r ( e . g . r e d o r w h i t e ) t h a n b l a c k . W e r e g a r d s u c h a h u b a s

a c a n d i d a t e o f c h a n c e , b e c a u s e i t c a n b e m e a n i n g f u l f o r a d e c i s i o n t o j u m p

f r o m a n i s l a n d t o a n o t h e r i s l a n d .


15/48



F i g . 4 s u p p o r t s t h e g e n e r a t i o n o f a s c e n a r i o o f c r i m i n a l b e h a v i o r s , s u c h a s t h e

o n e b e l o w , b y r e c o l l e c t i n g i n f o r m a t i o n a b o u t t h e m e m b e r s f r o m e x p l i c i t o r i m p l i c i t

( t a c i t ) k n o w l e d g e o f i n t e l l i g e n c e a n a l y s t s .

\ M e m b e r 1 ; m e m b e r 2 , a n d m e m b e r 3 a r e w o r k i n g t o g e t h e r . A n d ,

m e m b e r ; m e m b e r 6 , a n d m e m b e r 7 f o r m a n o t h e r g r o u p . W h e n t h e y

m e e t m e m b e r 9 m e m b e r 9 m a y g i v e c o m m a n d s t o b o t h g r o u p s f r o m

a h i g h e r l e v e l o f t h e o r g a n i z a t i o n . "

T h e a p p e a r a n c e o f a b r i d g i n g m e m b e r c a n b e a c e n t r a l t o p i c i n t h e a n a l y s t s ' c o m -

m u n i c a t i o n a b o u t c r i m e s , a n d a i d s u s e r ' s n d i n g o f c h a n c e e v e n t s o r i t e m s .

F i g . i s t h e K e y G r a p h , f o r D 2 i n E q . ( 6 . ) , t h e i n t e r n a l d a t a f r o m a c o m m u n i c a -

t i o n o f i n t e l l i g e n c e a n a l y s t s a b o u t t h e c r i m i n a l g r o u p . E a c h w o r d i s r e g a r d e d h e r e

a s a n e v e n t , a n d a m e s s a g e f r o m o n e p a r t i c i p a n t a s a n e v e n t - s e t ( i . e . , a s o n e l i n e

i n E q . ( 6 . 2 ) ) . T h e l a r g e i s l a n d s i n F i g . , i . e . , f m e m b e r 1 , m e m b e r 2 , m e m b e r 3 g a n d

f m e m b e r , m e m b e r 6 , m e m b e r 7 g m e a n t h e t w o g r o u p s a r e f a m i l i a r t o t h e a n a l y s t s .

T h e b r i d g e s o f \ m e s s a g e " a n d \ f o r w a r d s " l i n k e d t o m e m b e r 9 s h o w t h a t m e m b e r 9

c a n j u s t f o r w a r d m e s s a g e s f r o m o n e g r o u p t o t h e o t h e r . O n t h e o t h e r h a n d , w e a l s o

n d i n F i g . t h a t m e m b e r 9 m a y b e a l e a d e r i f m e m b e r 4 i s \ s u p p o s e d " t o b e t h e

s e c r e t a r y . M r . Z d e c i d e d t o c h e c k t h e p e r s o n a l d a t a o f m e m b e r 4 , a s t h e \ o t h e r " c a n -

d i d a t e f o r b e i n g t h e l e a d e r . H o w e v e r , f r o m F i g . , M r . X a n d M r . Y s h o u l d n o t e t h a t

M r . Z w a s \ s u r e " t h a t m e m b e r 4 i s t h e s e c r e t a r y . T h e y s h o u l d n o w c h e c k w h y M r .

Z m a d e s u c h c o n t r a d i c t o r y c o m m e n t s . H e m a y b e t e l l i n g a l i e , o r m a y b e m e m b e r

4 i s u s u a l l y b e h a v i n g a m b i g u o u s l y . T h u s t h e f o c u s o f u n c e r t a i n t y i s d e t e c t e d , a n d

d a t a c a n b e c o l l e c t e d i n o r d e r t o i n c r e a s e t h e g r a n u l a r i t y o f i n f o r m a t i o n a b o u t t h e

u n c e r t a i n m e m b e r . I t i s p o t e n t i a l l y p o s s i b l e n o w t o d e c i d e t o p e r f o r m a n e w a c t i o n

f o r i n t e l l i g e n c e a n a l y s i s .

D 2 = t h e f o l l o w i n g t e x t : ( 6 . )

\ M r . X : m e m b e r 1 , m e m b e r 2 , a n d m e m b e r 3 a r e w o r k i n g t o g e t h e r .

M r . Y : A n d , m e m b e r a n d m e m b e r 7 a l s o f o r m a n o t h e r g r o u p . I d o

n o t k n o w m e m b e r 4 . . .

M r . Z : I g u e s s m e m b e r 9 i s t h e l e a d e r o f t h e a l l g r o u p o f m e m b e r 1 ,

m e m b e r 2 , m e m b e r 3 , m e m b e r , m e m b e r 6 , a n d m e m b e r 7 . I a m s u r e

m e m b e r 4 i s t h e i r s e c r e t a r y .

M r . X : I t h i n k m e m b e r , m e m b e r 6 , a n d m e m b e r 7 a r e a g r o u p .

B u t m e m b e r 9 f o r w a r d s t h e m e s s a g e f r o m m e m b e r 1 , m e m b e r 2 , a n d

m e m b e r 3 , t o m e m b e r , m e m b e r 6 , a n d m e m b e r 7 .

M r . Y : S u p p o s e m e m b e r 4 i s a s e c r e t a r y , w h o o t h e r t h a n m e m b e r 9

c a n b e t h e l e a d e r ? ?

M r . Z : L e t m e c h e c k t h e p e r s o n a l d a t a o f m e m b e r 4 a g a i n . "


16/48


1 0 Y u k i o O h s a w a

F i g . 3 . I n t e l l i g e n c e a n a l y s i s s e e k i n g h i d d e n l e a d e r .

F i g . 4 . A n e x a m p l e o f K e y G r a p h : I s l a n d s a r e o b t a i n e d f r o m D 1 i n E q . ( 6 . 2 ) , i n c l u d i n g s e t s

f m e m b e r 1 , m e m b e r 2 , m e m b e r 3 g a n d f m e m b e r 5 , m e m b e r 6 , m e m b e r 7 g r e s p e c t i v e l y . T h e n o d e s i n

a n d o u t s i d e o f t h e i s l a n d s s h o w f r e q u e n t a n d r a r e i t e m s r e s p e c t i v e l y , a n d m e m b e r 4 a n d m e m b e r 9

s h o w r a r e h u b s b r i d g i n g i s l a n d s .

F i g . 5 . K e y G r a p h , f o r t h e i n t e r n a l d a t a . I s l a n d s a r e o b t a i n e d f r o m D 2 i n E q . ( 4 ) .


17/48


D a t a C r y s t a l l i z a t i o n B e y o n d C h a n c e D i s c o v e r y 1 1

7 . D a t a C r y s t a l l i z e r a n d T h e D a t a C r y s t a l l i z a t i o n P r o c e s s

7 . 1 . D a t a C r y s t a l l i z e r : A T o o l f o r C r e a t i n g D u m m y I t e m s

D a t a C r y s t a l l i z a t i o n a i m s a t p r e s e n t i n g t h e h i d d e n s t r u c t u r e a m o n g e v e n t s i n c l u d i n g

u n o b s e r v a b l e o n e s . T h i s i s r e a l i z e d o n t h e p r o c e s s o f C h a n c e D i s c o v e r y , w i t h u s i n g a

t o o l c a l l e d D a t a C r y s t a l l i z e r , w h i c h i n s e r t s d u m m y i t e m s r e p r e s e n t i n g t h e p o t e n t i a l

e x i s t e n c e o f u n o b s e r v a b l e e v e n t s , t o t h e g i v e n d a t a . U n o b s e r v a b l e e v e n t s a n d t h e i r

r e l a t i o n s w i t h o t h e r e v e n t s a r e t o b e v i s u a l i z e d b y a p p l y i n g K e y G r a p h , i t e r a t i v e l y

t o t h e d a t a , w h i c h w e r e r e v i s e d b y i n s e r t i n g d u m m y i t e m s w i t h D a t a C r y s t a l l i z e r . I n

e a c h i t e r a t i o n , t h e s i z e o f e a c h i s l a n d i s i n c r e a s e d f o r r e d u c i n g t h e g r a n u l a r i t y o f t h e

s t r u c t u r e v i s u a l i z e d . I n e s s e n c e , D a t a C r y s t a l l i z e r w e d e v e l o p e d r u n s t h e f o l l o w i n g

p r o c e d u r e .

T h e p r o c e d u r e o f d a t a c r y s t a l l i z e r

k : = 1 ; H i d d e n 0 : = f g ; l i n e 0 : = f g ; M

1

: = a v a l u e p r o v i d e d b y t h e u s e r ;

f o r M

2

= 1 t o M

1

( M

1

+ 1 ) / 2 d o

f o r a l l i j 2 0 , 1 , 1 1 1 N s u c h t h a t j i d o

i f l i n e i a n d l i n e j a r e e q u a l t h e n i n s e r t ( D k i j ) ;

H : = k e y g r a p h ( D M

1

M

2

M

3

: = M

1

/ 2 ) ;

f o r j = 1 t o N d o

I f j 2n H t h e n d l e t e ( D k j ) ;

I f H 6= H i d d e n k t h e n

k : = k + 1 ;

H i d d e n k : = H ;

f o r m = 0 t o k 0 1 d o

d e l e t e ( D ; m ; H i d d e n m \ H ) ;

H i d d e n m : = H i d d e n m n H ;

L e t m e i n t r o d u c e t h e s y m b o l s e m p l o y e d : D i s t h e d a t a t o b e a n a l y z e d w i t h

K e y G r a p h i n t h e f u n c t i o n K e y G r a p h ( D M

1

M

2

M

3

) N i s t h e n u m b e r o f l i n e s

( c o - o c c u r r e n c e u n i t s ) i n t h e d a t a , a n d l i n e j r e p r e s e n t s t h e s e t o f i t e m s i n t h e j - t h

l i n e . H r e p r e s e n t s t h e s e t o f l i n e - n u m b e r s w h e r e t h e d u m m y i t e m s , w h i c h a p p e a r e d

o n t h e b r i d g e s o f t h e c u r r e n t K e y G r a p h , a r e p o s i t i o n e d i n t h e d a t a . H i d d e n i m e a n s

t h e s e t o f l i n e - n u m b e r s w i t h a d u m m y i t e m w h i c h a p p e a r e d o n a b r i d g e o f t h e

K e y G r a p h i n t h e i - t h l e v e l . T h e f u n c t i o n i n s e r t ( D k i j ) m e a n s t o i n s e r t k j

t h e d u m m y n o d e f o r t h e j - t h l i n e i n t h e k - t h l e v e l o f c r y s t a l l i z a t i o n , t o t h e i - t h l i n e

o f d a t a D a n d f r o m d a t a D d e l e t e ( D k j ) m e a n s t o d e l e t e k j , t h e d u m m y i t e m

f o r t h e j - t h l i n e o n t h e k - t h l e v e l , f o r a l l i t s a p p e a r a n c e s i n d a t a D

I n t u i t i v e l y , w e c a n e x p l a i n t h e p r o c e d u r e a s f o l l o w s . C r y s t a l l i z a t i o n h e r e m e a n s

t o p r e s e n t t h e s t r u c t u r e o f t h e r e l a t i o n s h i p a m o n g i t e m s i n a n d o u t o f ( d u m m y )

t h e d a t a . F i r s t , k , t h e l e v e l o f c r y s t a l l i z e d s t r u c t u r e , i s s e t t o 1 . T h e v a l u e o f M

1

( t h e n u m b e r o f b l a c k n o d e s i n K e y G r a p h ) i s d e n e d b y t h e u s e r ( s ) . T h e n , M

2

( t h e


18/48



n u m b e r o f b l a c k l i n e s ) i s i n c r e m e n t e d f r o m 1 , u n t i l a l l t h e n o d e s i n t h e o r i g i n a l d a t a

a r e c o n n e c t e d a n d f o r m a s i n g l e i s l a n d .

F o r e a c h v a l u e o f M

2

, d u m m y i t e m s a r e i n s e r t e d i n t o D . T h e t h i r d a n d t h e f o r t h

l i n e s o f t h e p r o c e d u r e a b o v e m e a n : I f 2 o r m o r e l i n e s h a v e t h e s a m e s e t o f i t e m s ,

t h e s a m e d u m m y i t e m i s i n s e r t e d t o a l l t h o s e l i n e s , s u x e d w i t h t h e l i n e - n u m b e r

o f t h e r s t o f t h o s e l i n e s . T h a t i s , k j i s i n s e r t e d t o t h e j - t h l i n e , a n d , i f t h e r e i s a

l i n e ( t h e i - t h l i n e ) o f t h e s a m e s e t o f i t e m s a s i n t h e j - t h l i n e , k j i s i n s e r t e d t o a l l

t h o s e l i n e s .

T o t h i s d a t a w i t h i n s e r t e d d u m m y n o d e s , K e y G r a p h i s a p p l i e d a s i n t h e f t h l i n e .

T h e n , t h e n e w e s t d u m m y i t e m s w h i c h d i d n o t a p p e a r o n t h e b r i d g e s o f K e y G r a p h

a r e d e l e t e d f r o m D a s i n t h e s i x t h a n d t h e s e v e n t h l i n e s . T h e i n t e g e r k , t h e l e v e l o f

c r y s t a l l i z e d s t r u c t u r e , i s i n c r e m e n t e d i f H , t h e s e t o f d u m m y n o d e s i n t h e o b t a i n e d

K e y G r a p h , d i e r s f r o m H i d d e n k i . e . t h e s e t o f t h e l a t e s t d u m m y i t e m s o b t a i n e d

s o f a r . I f a l i n e i n t h e d a t a i n c l u d e s 2 o r m o r e d u m m i e s , a l l t h e d u m m y i t e m s i n

t h e l i n e e x c e p t f o r t h e h i g h e s t l e v e l a r e d e l e t e d , a s i n t h e e l e v e n t h t o t h e t h i r t e e n t h

l i n e s i n t h e p r o c e d u r e .

A f t e r a l l , t h e f o l l o w i n g a r e o b t a i n e d :

1 ) A n e w d a t a s e t w i t h d u m m y i t e m s , c o r r e s p o n d i n g t o h i d d e n e v e n t s t h a t

c o n n e c t s u b s t r u c t u r e s i n e a c h l e v e l .

2 ) k e y g r a p h ( D M

1

M

2

M

3

) f o r t h e o b t a i n e d d a t a D , f o r a r b i t r a r i l y d e -

t e r m i n e d v a l u e s o f M

1

M

2

, a n d M

3

. B y i n c r e a s i n g M

2

, w e c a n f o c u s t h e

o u t p u t t o t h e h i g h e r l e v e l o f t h e h i d d e n s t r u c t u r e . B y d e c r e a s i n g M

2

, t h e

g r a n u l a r i t y o f t h e v i s u a l i z e d s t r u c t u r e i s i n c r e a s e d .

D a t a C r y s t a l l i z a t i o n w o r k s i n t h e w a y l i k e t h e c r y s t a l l i z a t i o n o f s n o w . A c r y s t a l -

l i z i n g i t e m o f t h e d a t a p l a y s a r o l e l i k e a p a r t i c l e o f d u s t , w h i c h c o n n e c t s m o l e c u l e s

o f w a t e r i n a c o l d t e m p e r a t u r e a n d f o r m s a s n o w c r y s t a l . T h e i n c r e a s e i n M

2

c o r -

r e s p o n d s t o t h e d e c r e a s e i n t e m p e r a t u r e , s o t h e g r a d u a l i n c r e a s e i n M

2

l e a d s t o a

w e l l - s t r u c t u r e d K e y G r a p h c o r r e s p o n d i n g t o a w e l l - s t r u c t u r e d s n o w c r y s t a l o b t a i n e d

f r o m g r a d u a l c o o l i n g o f a i r .

7 . 2 . T h e H u m a n - M a c h i n e I n t e r a c t i o n i n D a t a C r y s t a l l i z a t i o n

T h e t o o l D a t a C r y s t a l l i z e r s h o u l d w o r k i n S t e p 3 ) o f t h e D o u b l e H e l i x p e o c e s s a s

d e s c r i b e d i n t h e l i s t b e l o w , b e c a u s e D a t a C r y s t a l l i z a t i o n i s a k i n d o f C h a n c e D i s -

c o v e r y . T h a t i s , D a t a C r y s t a l l i z a t i o n s e r v e s t h e u n d e r s t a n d i n g o f d e e p - l e v e l c h a n c e

e v e n t s , b u t t h e d u m m y i t e m s c o r r e s p o n d i n g t o t h e s e e v e n t s c a n n o t b e u n d e r s t o o d

i f t h e u s e r i s s t i l l i n a n e a r l y s t a g e o f C h a n c e D i s c o v e r y . T h e r e i s a r i s k o f d i s t u r b i n g

u s e r ' s u n d e r s t a n d i n g i f a t o o c o m p l e x s t r u c t u r e i s s h o w n t o s o m e o n e w h o s e e k s s i m -

p l e i n f o r m a t i o n . T h u s , D a t a C r y s t a l l i z e r w o r k s o n l y i f t h e u s e r i s c o n c e r n e d w i t h

u n o b s e r v a b l e l e v e l o f t h e s t r u c t u r e :

T h e R e n e d D H p r o c e s s f o r D a t a C r y s t a l l i z a t i o n

S t e p 1 ) E x p r e s s t h e u s e r ' s ( o r t h e u s e r s g r o u p ) o w n c o n c e r n w i t h a c h a n c e .


19/48



S t e p 2 ) O b t a i n t h e e x t e r n a l d a t a , i . e . , t h e d a t a f r o m t h e t a r g e t e n v i r o n m e n t , r e l -

e v a n t t o t h e c u r r e n t c o n c e r n .

S t e p 3 ) P r o p o s e s c e n a r i o s f r o m t h e t h o u g h t s o f u s e r ( s ) b y l o o k i n g a t t h e s c e n a r i o

m a p , w h i c h i s t h e r e s u l t o f v i s u a l d a t a m i n i n g w i t h a t o o l s u c h a s K e y G r a p h ,

a p p l i e d t o t h e e x t e r n a l d a t a o b t a i n e d i n S t e p 2 . I f t h e p a r t i c i p a n t s w a n t

t o i n v e s t i g a t e u n o b s e r v a b l e l e v e l s o f t h e s t r u c t u r e , u s e D a t a C r y s t a l l i z e r .

O t h e r w i s e u s e K e y G r a p h w i t h o u t i n s e r t i n g d u m m y i t e m s .

S t e p 4 ) V i s u a l i z e t h e i n t e r n a l d a t a , i . e . , t h e d o c u m e n t e d t h o u g h t s o f u s e r ( s ) i n

S t e p 3 , b y v i s u a l t e x t m i n i n g .

S t e p 5 ) C h o o s e t h e o p t i m a l s c e n a r i o ( b y d i s c o v e r i n g c h a n c e s i f a n y ) , f r o m t h e

m a p s o f S t e p 3 a n d S t e p 4 .

S t e p 6 ) E v a l u a t e t h e s c e n a r i o o b t a i n e d i n S t e p ) f r o m t h e b e n e t / l o s s o f t h e o b -

t a i n e d s c e n a r i o , a n d g o t o S t e p 1 ) i f o n e o b t a i n s a n e w c o n c e r n f o r i m p r o v i n g

t h e s c e n a r i o .

8 . A R u n n i n g C a s e o f D a t a C r y s t a l l i z a t i o n

W e t o o k a s e r i e s o f m e e t i n g s i n a f a c u l t y o f 2 1 m e m b e r s , a s t h e t a r g e t d a t a t o

a n a l y z e . I n D a , a p a r t o f d a t a o n t h e p a r t i c i p a n t s a r e l i s t e d , o b t a i n e d i n S t e p 2 ) f o r

o u r c o n c e r n \ w h e r e i s t h e r e a l l e a d e r ? " H e r e , e a c h l i n e c o r r e s p o n d s t o o n e m e e t i n g

b y s o m e p a r t o f t h e f a c u l t y . N o t e t h a t t h e n a m e s a r e a r r a n g e d t o h i d e r e a l i n d i v i d u a l

n a m e s , i . e . , i f r e a d e r n d s a f a c u l t y o f s i m i l a r m e m b e r s , i t m i g h t n o t b e t h e c a s e

d e a l t w i t h h e r e .

D a = t s u b a k i s a r u o g u r a k u w a

t s u b a k i s a r u k u w a k a w a i

k a w a i k u w a n a g a i

o g u r a y o s h i d a t s u b a k i k a w a i x u

x u m a k i m o t o t s u b a k i y u j i

r y o k e n a g a i

( 8 . 6 )

F i g . 6 i s t h e r e s u l t o f K e y G r a p h i n S t e p 3 ) , f o r M

1

= 2 0 , M

2

= 2 0 , a n d M

3

= 2 0 ,

f r o m D a . E v e n t h o u g h K e y G r a p h s e a r c h e d 2 0 h u b s b r i d g i n g b e t w e e n i s l a n d s i n

t h i s s e t t i n g , w e n d a l l i s l a n d s s e p a r a t e d i . e . , n o b r i d g e s a m o n g t h e m . T h a t i s , t h e

f a c u l t y l o o k e d l i k e a s e t o f g r o u p s i r r e l e v a n t t o e a c h o t h e r , i n s p i t e o f t h e b r i d g i n g

f u n c t i o n o f K e y G r a p h . T h i s w a s u n r e a s o n a b l e , b e c a u s e t h e t e a m w o r k o f t h i s f a c u l t y

w a s g o o d e n o u g h t o c o m b i n e t h e k n o w l e d g e o f p r o f e s s o r s a n d m a k e c o l l a b o r a t i v e

p r o j e c t s . T h u s , w e c a m e t o i n v e s t i g a t e d e e p e r l e v e l s i n c l u d i n g h i d d e n e v e n t s . T h e

d u m m y n o d e s a r e n o w i n s e r t e d , d e n o t e d 1 x f o r t h e x - t h l i n e , t o o b t a i n D b b e l o w .


20/48



D b = t s u b a k i s a r u o g u r a k u w a 1 1

o s a w a y u j i y o s h i d a x u k a w a i s a n o 1 2

t s u b a k i s a r u k u w a k a w a i 1 3

k a w a i k u w a n a g a i 1 4

o g u r a y o s h i d a t s u b a k i k a w a i x u 1

x u m a k i m o t o t s u b a k i y u j i 1 6

r y o k e n a g a i 1 7

( 8 . 7 )

F i g . 7 i s t h e K e y G r a p h f o r D b . W e n o w n d t h a t s o m e d u m m y n o d e s r e m a i n i n g

i n t h e g r a p h , f o r m i n g t h e b r i d g e s a m o n g i s l a n d s . F o r e x a m p l e , w e n d d u m m y 1

b e t w e e n y o s h i d a a n d o g u r a . T h i s m e a n s s o m e h i d d e n i t e m r e l e v a n t t o t h e f t h

m e e t i n g ( t h e f t h l i n e i n E q . ( 8 . 7 ) ) m a d e a s i g n i c a n t b r i d g e f o r t h e s t r u c t u r e o f t h e

f a c u l t y . A l l d u m m y i t e m s w h i c h d i d n o t a p p e a r a s b r i d g e s i n F i g . 7 a r e d e l e t e d f r o m

t h e d a t a ( s e e t h e s i x t h a n d t h e s e v e n t h l i n e s i n t h e p r o c e d u r e o f D a t a C r y s t a l l i z e r ) .

F i g . 6 . T h e o r i g i n a l K e y G r a p h f o r m e m b e r s o f a g r o u p .

T h e n , n e w d u m m y n o d e s 2 x f o r t h e s e c o n d l e v e l a r e i n s e r t e d t o o b t a i n D c i n

E q . ( 8 . 8 ) . H o w e v e r , l e t u s s k i p t h e o u t p u t o f K e y G r a p h f o r D c a n d j u s t s h o w t h e

c h a n g e i n t h e d a t a . T h a t i s , d u m m y n o d e s i n t h e s e c o n d l e v e l a r e d e l e t e d i f t h e y d o

n o t a p p e a r i n t h e r e s u l t a n t K e y G r a p h , a n d t h e d a t a c h a n g e i n t o D d i n E q . ( 8 . 9 ) .

H a v i n g t h e t o o l r u n i n t h i s w a y t o t h e t h i r d l e v e l , D e a s i n E q . ( 8 . 1 0 ) i s o b t a i n e d .


21/48



D c = t s u b a k i s a r u o g u r a k u w a 1 1 2 1

o s a w a y u j i y o s h i d a x u k a w a i s a n o 1 2 2 2

t s u b a k i s a r u k u w a k a w a i 1 3 2 3

k a w a i k u w a n a g a i 2 4

o g u r a y o s h i d a t s u b a k i k a w a i x u 1 2

x u m a k i m o t o t s u b a k i y u j i 2 6

r y o k e n a g a i 1 7 2 7 ( 8 . 8 )

D d = t s u b a k i s a r u o g u r a k u w a 1 1







r y o k e n a g a i t s u b a k i 1 7 ( 8 . 9 )

D e = t s u b a k i s a r u o g u r a k u w a 1 1







r y o k e n a g a i t s u b a k i 1 7

( 8 . 1 0 )

F i g . 8 i s t h e r e s u l t f o r D e , w i t h M

2

i n c r e a s e d u p t o 3 0 . I n c r e a s i n g t h e n u m b e r o f

b l a c k l i n k s ( M

2

) m e a n s t o e n l a r g e i s l a n d s , f o r i g n o r i n g t h e l o c a l s t r u c t u r e b e t w e e n

s m a l l i s l a n d s , a n d t o f o c u s a t t e n t i o n o n t h e h i g h e r l e v e l . S o m e d u m m y n o d e s i n t h e

s a m e l i n e a p p e a r i n t h e s a m e p o s i t i o n i n t h e g r a p h , s u c h a s d u m m y 1 2 a n d d u m m y

3 2 i n F i g . 8 . I n s u c h a c a s e , o n l y d u m m y 3 2 s h o u l d r e m a i n h e r e , s o d u m m y 1 2 i s

d e l e t e d f r o m t h e d a t a s e t a s i n t h e t e n t h t o t h e t w e l f t h l i n e s i n t h e p r o c e d u r e o f

D a t a C r y s t a l l i z e r .

A f t e r o b t a i n i n g D e , t h e i n f o r m a t i v e d a t a w i t h u n o b s e r v a b l e e v e n t s , w e c a n r e -

d u c e t h e n u m b e r o f b l a c k l i n e s , i . e . , M

2

, t o o b t a i n F i g . 9 t o s e e t h e l o w e r - l e v e l


22/48



( d u m m y 1 x ) , t h e m i d d l e - l e v e l ( d u m m y 2 x ) , a n d t h e h i g h - l e v e l ( d u m m y 3 x ) s t r u c -

t u r e s o f t h e h u m a n r e l a t i o n s i n t h e f a c u l t y . W e a p p a r e n t l y o b t a i n n e w e r n d i n g s

t h a n F i g . 6 . O n F i g . 9 , t h e t h o u g h t s o f s o m e f a c u l t y m e m b e r s w e r e c o l l e c t e d a s b e l o w .

T h e 3 x d u m m y n o d e s r e p r e s e n t t h e t o p l e v e l l i n k s . F o r e x a m p l e , O g u r a

w a s t h e h e a d o f t h e b i g g e s t d e p a r t m e n t i n t h e f a c u l t y t w o y e a r s a g o , a n d

h i s n o d e i s l i n k e d t o t h e d e a n . Y o s h i d a w o r k s i n c o m p u t e r s c i e n c e , a n d i s

t h e c u r r e n t h e a d o f t h e d e p a r t m e n t . O g u r a a n d Y o s h i d a a r e l i n k e d b y 3 .

T h e n e x t l e v e l ( 2 x ) d u m m y n o d e s c o n n e c t p a i r s e . g . f R y o k e , N a g a i g

W a t a n a b e , S a n o . T h e y w e r e d i s c u s s i n g t h e l o c a l a r r a n g e m e n t s o f d e p a r t -

m e n t s , i . e . , m i d d l e - c l a s s m a n a g e m e n t o f t h e f a c u l t y .

T h e n e x t l e v e l ( 1 x ) d u m m y n o d e s l i n k p a i r s s u c h a s f S a r u , K u w a g . T h e s e

c o r r e s p o n d t o p r o p o s a l s a n d a c c e p t a t i o n f r o m y o u n g s t a s u c h a s S a r u a n d

K u w a , i . e . , b o t t o m u p p r o p o s a l s .

( c o n t i n u i n g t o o t h e r m e s s a g e s . . . )

T h e s e m e s s a g e s c o n s t i t u t e t h e i n t e r n a l d a t a u s e d i n S t e p 4 ) , i n t h e R e n e d D H

P r o c e s s f o r D a t a C r y s t a l l i z a t i o n . B y l o o k i n g a t F i g . 8 o b t a i n e d b y K e y G r a p h f o r t h e

i n t e r n a l d a t a , t h e p a r t i c i p a n t s c l e a r l y b e c a m e a w a r e t h a t t h e c o m m o n i n t e r e s t s o f

t h e d e a n ( n o t i n c l u d e d i n t h e d a t a o f m e e t i n g p a r t i c i p a n t s ) , a n d t h e p r e v i o u s a n d

t h e c u r r e n t h e a d s o f t h e b i g g e s t d e p a r t m e n t a r e i m p o r t a n t f o r t h e m a n a g e m e n t o f

t h e w h o l e f a c u l t y . B y l o o k i n g a t t h e c o m m o n o p i n i o n s o f t h e s e h e a d s , i t i s p o s s i b l e

t o d e t e c t s i g n s o f n e w t r e n d s o f t h i s f a c u l t y . I n e s s e n c e , t h e s a m e p r o d e c u r e a s t h e

o n e s h o w n i n t h i s e x a m p l e i s c o n s i d e r e d t o b e a p p l i c a b l e t o o t h e r h u m a n s o c i e t i e s ,

s u c h a s c r i m i n a l g r o u p s , c o n s u m e r s , r e s e a r c h e r s i n a s c i e n t i c d o m a i n , e t c .

9 . C o n c l u s i o n s

D a t a C r y s t a l l i z i n g m e a n s t o e x t e n d C h a n c e D i s c o v e r y t o t h e d i s c o v e r y o f s i g n i c a n t

e v e n t s i n m o r e u n c e r t a i n e n v i r o n m e n t t h a n w e h a v e b e e n d e a l i n g w i t h i n s t u d i e s o n

C h a n c e D i s c o v e r y . A n d , t h e s p h e r e o f r e a l w o r l d a p p l i c a t i o n s l i n k e d f r o m t h i s b a s i c

r e s e a r c h i s e x p e c t e d t o i n c l u d e i n t e l l i g e n c e a n a l y s i s , d e v e l o p m e n t o f n e w p r o d u c t s ,

a i d i n g c o r p o r a t e b e h a v i o r s b y d e t e c t i n g i n t e r e s t o f e m p l o y e e s , e t c .

A r e l e v a n t r e s e a r c h a r e a t o C h a n c e D i s c o v e r y i s E v i d e n c e E x t r a c t i o n a n d

L i n k D i s c o v e r y ( E E L D ) , w h e r e i m p o r t a n t l i n k s o f p e o p l e w i t h o t h e r p e o p l e a n d

w i t h t h e i r o w n a c t i o n s a r e t o b e d i s c o v e r e d f r o m h e t e r o g e n e o u s s o u r c e s o f d a t a

1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1

. T h e d i e r e n c e b e t w e e n C h a n c e D i s c o v e r y a n d E E L D , f o r t h e

t i m e b e i n g , i s i n t h e p o s i t i o n o f h u m a n f a c t o r s i n t h e r e s e a r c h a p p r o a c h e s . I n C h a n c e

D i s c o v e r y , t h e v i s u a l i z a t i o n t e c h n i q u e s s u c h a s K e y G r a p h h a v e b e e n u s e d f o r c l a r -

i f y i n g t h e e e c t o f c h a n c e s , b y a c t i v a t i n g u s e r ' s t h o u g h t s o n s c e n a r i o s i n t h e r e a l

e n v i r o n m e n t . O n t h e o t h e r h a n d , t h e E E L D p r o g r a m m a i n l y c o n t r i b u t e d t o i d e n t i -

f y i n g t h e m o s t s i g n i c a n t l i n k s a m o n g i t e m s m o r e a u t o m a t i c a l l y a n d p r e c i s e l y t h a n

h u m a n .

S t u d i e s o n E E L D a r e c o m i n g t o b e o r i e n t e d t o c o u p l i n g s y m b o l i c e x p r e s s i o n s o f


23/48



h u m a n k n o w l e d g e w i t h a m a c h i n e l e a r n i n g s y s t e m

2 0

, a n d a l s o i n t r o d u c i n g t h e u s e

o f d a t a v i s u a l i z a t i o n f o r d e c i s i o n m a k i n g

1 7 1 8

. O n t h e o t h e r h a n d , C h a n c e D i s c o v e r y

h a s b e e n i n t e g r a t i n g t h e h u m a n p r o c e s s o f e x t e r n a l i z i n g t h e t a c i t e x p e r i e n c e s w i t h

t h e p o w e r o f m a c h i n e s f o r n d i n g a s u r p r i s i n g t r i g g e r t o n e w a c t i o n s i n t h e r e a l

e n v i r o n m e n t . T h a t i s , h u m a n ' s i n t e r a c t i o n w i t h m a c h i n e i n t e l l i g e n c e i s c o m i n g t o

t h e c e n t e r s o f t h e s e t w o d o m a i n s .

W e n a l l y p r e d i c t t h e m e e t i n g p o i n t o f C h a n c e D i s c o v e r y a n d E E L D w i l l b e t h e

d e t e c t i o n o f u n o b s e r v e d b u t s i g n i c a n t e v e n t s , a s i n t h e c h a l l e n g e o f D a t a C r y s -

t a l l i z a t i o n . A s s h o w n i n t h e j u m p f r o m F i g . 9 t o F i g . 1 0 , t h e c l a r i c a t i o n o f h i d d e n

l i n k s v i a u n o b s e r v a b l e e v e n t s a r e n a l l y u p t o t h e h u m a n t h o u g h t . H u m a n s h o u l d

l o o k i n t o m o r e a n d m o r e g r a n u l a r i n f o r m a t i o n a b o u t t h e e n v i r o n m e n t , h a n d i n h a n d

w i t h t h e c r y s t a l l i z a t i o n o f K e y G r a p h . T h i s i s l i k e a s c i e n t i s t i n a l a b o r a t o r y c o o l i n g

t h e t e m p e r a t u r e s l o w l y , c a r e f u l l y m o n i t o r i n g t h e e x p e r i m e n t a l c o n d i t i o n , i n o r d e r

t o o b t a i n a w e l l - s t r u c t u r e d c r y s t a l .

R e f e r e n c e s

1 . O h s a w a , Y . , M c B u r n e y , P . ( e d s ) , C h a n c e D i s c o v e r y ( S p r i n g e r V e r l a g , H e i d e l b e r g ,

2 0 0 3 )

2 . A b e , A . , O h s a w a , Y . ( e d s ) , R e a d i n g s i n C h a n c e D i s c o v e r y ( A d v a n c e d K n o w l e d g e I n -

t e r n a t i o n a l , A u s t r a l i a , 2 0 0 5 )

3 . T h e C h a n c e D i s c o v e r y C o n s o r t i u m ( C D C ) , E x a m p l e s o f C h a n c e D i s c o v e r y ,

h t t p : / / w w w . c h a n c e d i s c o v e r y . c o m ( 2 0 0 4 )

4 . G a v e r W . W . , B e a v e r J . , a n d B e n f o r d S . , 2 0 0 3 , A m b i g u i t y a s a R e s o u r c e f o r D e s i g n ,

i n P r o c e e d i n g s o f C o m p u t e r H u m a n I n t e r a c t i o n s

5 . T h e D a n i s h B o a r d o f T e c h n o l o g y , 2 0 0 3 , E u r o p e a n P a r t i c i p a t o r y T e c h n o l o g y A s s e s s -

m e n t : P a r t i c i p a t o r y M e t h o d s i n T e c h n o l o g y A s s e s s m e n t a n d T e c h n o l o g y D e c i s i o n -

M a k i n g , . h t t p : / / w w w . t e k n o . d k / e u r o p t a

6 . J o s h i , M . , K u m a r , V . , A g a r w a l , R . E v a l u a t i n g B o o s t i n g A l g o r i t h m s t o C l a s s i f y R a r e

C l a s s e s : C o m p a r i s o n a n d I m p r o v e m e n t s , I n P r o c . o f T h e F i r s t I E E E I n t e r n a t i o n a l

C o n f e r e n c e o n D a t a M i n i n g , ( S a n J o s e , 2 0 0 1 )

7 . W e i s s , G M . , a n d H i r s h , H ( 1 9 9 8 ) . L e a r n i n g t o P r e d i c t R a r e E v e n t s i n E v e n t S e q u e n c e s ,

I n P r o c e e d i n g s o f t h e F o u r t h I n t e r n a t i o n a l C o n f e r e n c e o n K n o w l e d g e D i s c o v e r y a n d

D a t a M i n i n g ( K D D - 9 8 ) , ( A A A I P r e s s , M e n l o P a r k , 1 9 9 8 ) p p . 3 5 9 { 3 6 3

8 . O h s a w a , Y . , a n d U s u i , M . , : W o r k s h o p w i t h T o u c h a b l e K e G r a p h A c t i v a t i n g T e x t i l e

M a r k e t , A b e , A a n d O h s a w a , Y ( e d s ) R e a d i n g s i n C h a n c e D i s c o v e r y ( A d v a n c e d

K n o w l e d g e I n t e r n a t i o n a l , A u s t r a l i a , 2 0 0 5 ) p p . 3 8 5 { 3 9 4

9 . O h s a w a Y , F u j i e H , S a i u r a A , O k a z a k i N , a n d M a t s u m u r a N , 2 0 0 4 , P r o c e s s t o D i s -

c o v e r i n g I r o n D e c r e a s e a s C h a n c e t o U s e I n t e r f e r o n t o H e p a t i t i s B , i n P a t o n , R . ( e d )

M u l t i d i s c i p l i n a r y A p p r o a c h e s t o T h e o r y i n M e d i c i n e ( E l s e v i e r , T h e N e t h e r l a n d , 2 0 0 5 )

1 0 . O h s a w a , Y . , S o m a , H . , M a t s u o , Y . , U s u i , M . , a n d M a t s u m u r a , N . , F e a t u r i n g W e b

C o m m u n i t i e s b a s e d o n W o r d C o - o c c u r r e n c e S t r u c t u r e o f C o m m u n i c a t i o n s , P r o c e e d -

i n g s o f t h e E l e v e n t h C o n f . W o r l d W i d e W e b ( W W W 1 1 ) , ( A C M p r e s s , N e w Y o r k ,

2 0 0 2 )

1 1 . O h s a w a Y , 2 0 0 3 b , K e y G r a p h : V i s u a l i z e d S t r u c t u r e A m o n g E v e n t C l u s t e r s , i n O h s a w a

Y a n d M c B u r n e y P . e d s , C h a n c e D i s c o v e r y , ( S p r i n g e r V e r l a g , 2 0 0 3 ) p p . 2 6 2 { 2 7 5

1 2 . O h s a w a , Y . , B e n s o n , N . E . , a n d Y a c h i d a , M . , K e y G r a p h : A u t o m a t i c I n d e x i n g b y C o -

o c c u r r e n c e G r a p h b a s e d o n B u i l d i n g C o n s t r u c t i o n M e t a p h o r , P r o c . A d v a n c e d D i g i t a l


24/48



L i b r a r y C o n f e r e n c e ( I E E E A D L ' 9 8 ) , ( I E E E p r e s s , L o s A l a m o s , 1 9 9 8 ) , p p . 1 2 { 1 8

1 3 . O h s a w a , Y . , 2 0 0 3 a , M o d e l i n g t h e P r o c e s s o f C h a n c e D i s c o v e r y , O h s a w a , Y . a n d

M c B u r n e y e d s , C h a n c e D i s c o v e r y ( S p r i n g e r V e r l a g , H e i d e l b e r g , 2 0 0 3 ) p p . 2 { 1 5

1 4 . O h s a w a , Y . : K e y G r a p h a s R i s k E x p l o r e r f r o m E a r t h q u a k e S e q u e n c e , J o u r n a

Chance Discovery With Data Crystallization - Discovering Unobservable Events_2006

Documents

Transcript of Chance Discovery With Data Crystallization - Discovering Unobservable Events_2006