Data Science Center Eindhoven - 4TU.Federation · 4TU AMI SRO Big Data Meeting Big Data:...
Transcript of Data Science Center Eindhoven - 4TU.Federation · 4TU AMI SRO Big Data Meeting Big Data:...
4TU AMI SRO Big Data MeetingBig Data Mathematics in ActionNovember 24 2017
Data Science Center Eindhoven
The Mathematics Behind Big Data
Alessandro Di Bucchianico
Outline
bull ldquoBig Datardquobull Some real-life examples with ldquohiddenrdquo mathematicsbull Some mathematical developmentsbull Conclusions
What is big data
Term coined by John Mashey chief scientist at Silicon Graphics in the 1990rsquos
hellipI was using one label for a range of issues and I wanted the simplest shortest phrase to convey that the boundaries of computing keep advancinghellip
But chemometrics has a long historyof analyzing ldquolargerdquo data sets
Four Vrsquos of Big Data
High-dimensional data ldquo119951119951 ≪ 119953119953rdquo
Topic Google Search
Topic Google Search
bull How can Google search so fast on its over 250 billion pages
bull What are proper ways to rank web pages
bull First idea count words in pages
Topic Google Search
Model the WWW as a graphbull Squares (nodes vertices) denote web pagesbull Arrows (edges) denote links from one page to another
Relevance is translated into number of incoming links weighted by importance of referring pages
Topic Google Search
Topic Google Search
A =
Topic Google Search
Equivalent Random Surfer Model
bull Start at any point on the graphbull Assign equal probabilities to all outgoing linksbull Choose new node with probabilities given by the
linksbull Relevance is proportion of visits after surfing for a
very long time (ldquostationary distributionrdquo)
Google Matrix
bull Choose a ldquodamping factorrdquo p (0ltplt1) bull Googlersquos p is secret but around 015
Interpretation for random surferbull choose next link according to A with probability 1- pbull ldquoteleportrdquo to random page with probability p
119872119872 = 1 minus 119901119901 119860119860 + 119901119901119901119901
119901119901 =1119899119899
1 ⋯ 1⋮ ⋱ ⋮1 ⋯ 1
Choice of Matrix B
Google choose other B to avoid unwanted boosting of pages
Practical Issues
bull The real Google matrix has size in the order of 30 billion columns x 30 billion rows
bull Every day around 250000 new pages are added bull Matrix has many zeros (100 links to a page)
which speeds up calculationsbull Doing the matrix calculation
requires fast computers butalso advanced math
Topic Streaming algorithms
High volume high velocity data makes exact counting of frequencies or number of different items practically infeasible but approximate answers suffice in several applications (web site counters customer counts in retail transactionshellip)
New algorithms using advanced probabilistic methods (random hash functions)bull Count-Min Sketch bull MinHashbull HyperLogLog
Topic Uncertainty Quantification
Virtual prototyping using mathematical models UQ does not deal with1 unknown uncertainty in the initial
conditions of parameters2 parametrisation of design building blocks dimension
reduction
For 1) polynomial chaos (Wiener chaos) inverse statistical models Bayesian analysis (calibration)
For 2) Model Order Reduction
Topic Network monitoring
0
0002
0004
0006
0008
001
0012
1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Measu
re V
alu
e
Time
Al-Qaeda Network Measures
Betweeness
Density
Closeness
Challengesbull monitor high number of variablesbull models to capture structural changesbull scalable algorithms for likelihood ratio calculations
Closeness CUSUM Statistic for Al-Qaeda (1994-2004)
0
1
2
3
4
5
6
7
8
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
C+
Change Point
Signal