Visualization for Multimedia Contents

Takayuki ItohOchanomizu University, Japan

Shonan NII Meeting 2014/3/10

Itoh Laboratory, Ochanomizu University

This talk introduces our works on…

4

Itoh Laboratory, Ochanomizu University

Music browsingPhotograph browsing

[AVI 10, ACM SAC 11 etc][ACM SAC 11, IV 11 etc]

5

Itoh Laboratory, Ochanomizu University

Part I:Photograph browsing

Photograph browsing tools

• To browse large photo collection intuitively

6

Itoh Laboratory, Ochanomizu University

PhotoMesa Design Gallery

Photograph browsing tools

• To browse large photo collection intuitively• Personal or professional• Structured (tree, graph, time-sequence…) or scattered• Image processing algorithms

– Meta-data based: (time, place, person …)– Low-level feature based: (color, texture, shape …)– High-level recognition based: (human face, object, scene …)

7

Itoh Laboratory, Ochanomizu University

PhotoMesa Design Gallery

Our zooming interface for photo exploration

• Overview the photo groups first, then zoom in– [Zoom-out] display representative photos of groups– [Zoom-in] display each photos in the zoomed groups

8

Itoh Laboratory, Ochanomizu University

[IV08]

Zoom Out

Zoom In

Tree visualization for photo browsing

• Hierarchy representation by nested rectangles• Mapping photos onto icons (= leaves) (for zoom in)

• Mapping photos into rectangles (= branches) (for zoom out)

9

[TVCG04, CG&A06]

Itoh Laboratory, Ochanomizu University

Photo on icon

Photo in rectangle

Hierarchy representation

Photo collection as “life log”

• Many photos by travel, events, ...• Often we reflect our life from “When, Where, Who”

↓• Enjoyable life reflection by looking at photos

from the viewpoints of “When, Where, Who”

10

Visit to Kyoto every year

Hokkaido with family in 2007

Hawaii with friends in 2009

High school travel to Okinawa

Itoh Laboratory, Ochanomizu University

Itoh Laboratory, Ochanomizu University

Photo browser by “When, Where, Who”

• Photo display based on time and place

• Smooth interaction(Rotation, zoom)

• All-in-one display of faces in the photos

• Photo discovery by interaction among time, place, and face

11

Extracted facesY: Latitude

X: Longitude

Z: Time

Photo display

Rotation

Zoom in

Zoom in

[AVI 10, ACM SAC 11 etc]

Life log of a student (1/2)

12

Europe with family

Guam with friends

Domestic trip in lab.

Europe with family

Itoh Laboratory, Ochanomizu University

Lab.

Friends

FamilyFamily

Photograph owner

[AVI 10, ACM SAC 11 etc]

Life log of a student (2/2)

13

Mid. America with family

Japan & Korea trips in lab.

Europe with friends USA trip

in lab.Japan with

friends

Australia

Itoh Laboratory, Ochanomizu University

Friends

FamilyFamily

Lab.

[AVI 10, ACM SAC 11 etc]

14

Itoh Laboratory, Ochanomizu University

Part II:Music browsing

Music browsing tools in my lab

15

Itoh Laboratory, Ochanomizu University

Overview & summarization

Recommendation & preference analysis

Visual user interface

Music browsing tools

• Quick finding & understanding before listening• Goal & purpose

– [Daily use] search, recommendation, …– [Expert use] education, practice, performance, …

• Input data– Audio (WAV, MP3 …) or Note (MIDI …)

• Musical genre– Pops or Classics– With vocals or Without vocals

16

Itoh Laboratory, Ochanomizu University

Music browsing tools in my lab

17

Itoh Laboratory, Ochanomizu University

Overview & summarization

Recommendation & preference analysis

Visual user interface

Visualizing structure of orchestra music

• Summarization of classical music score which has tens of note tracks

• Coloring of note blocks based on melodies

18

Woodwind

Brasswind

Strings

Former Latter

Itoh Laboratory, Ochanomizu University

[IV 11]

Example: “Valse des fleurs”

1919

Melody Harmony Bass Others

Remark to Purple melody

Itoh Laboratory, Ochanomizu University

[IV 11]

Condensation (= Vertical compression)

• Reduce the number of tracks while preserving important melodies

20

Itoh Laboratory, Ochanomizu University

[IV 11]

Condensation (= Vertical compression)

• Reduce the number of tracks while preserving important melodies

21

Itoh Laboratory, Ochanomizu University

[IV 11]

Visual Music Recommendation

• Input evaluation→ Learn & recommend preferable tunes

22

Itoh Laboratory, Ochanomizu University

Plot = tune

Automaticallygenerated

playlist

Not yet Playing Negative Positive

Buttons (play, stop,positive, negative)

[VINCI11]

Recommendation patterns

• Users A&B have similar preferences– Analog sounds– Simple arrange

• User C has much different preference– Low inharmonicity

23

Itoh Laboratory, Ochanomizu University

Loudness Loudness Loudness

High/low ratio High/low ratio Inharmonicity

User A User B User C

[VINCI11]

Conclusion

• Photograph browsing– Zooming interface [Tree vis.]– Meta-data based structuring

• Music browsing– Summarizing scores [Time-varying vis.]– Recommendation with scatterplots [Multi-dimensional vis.]

24

Itoh Laboratory, Ochanomizu University

References

Hierarchical Data Visualization Using a Fast Rectangle-Packing Algorithm, TVCG, 10(3), 302-313, 2004.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE13.pdfHierarchical Visualization of Network Intrusion Detection Data in the IP Address Space, CG&A, 26(2), 40-47, 2006.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE14.pdfCAT: A Hierarchical Image Browser Using a Rectangle Packing Technique, International Conference on Information Visualisation (IV08), 82-87, 2008.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE37.pdfA Personal Photograph Browser for Life Log Analysis based on Location, Time, and Person, ACM Symposium on Applied Computing, 1250-1257, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE58.pdfColorscore - Visualization and Condensation of Structure of Classical Music, International Conference on Information Visualisation (IV2011), 420-425, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE65.pdfMusiCube: A Visual Music Recommendation System featuring Interactive Evolutionary Computing, Visual Information Communication - Information Symposium (VINCI'11), 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE67.pdf

25

Itoh Laboratory, Ochanomizu University