Presentation at Nowcasting Symposium, Design/Media Arts Department, UCLA, Los Angeles, October 16-17, 2009(original title: “Cultural Analytics annual report 6/2008-9/2009)

Dr. Lev ManovichDirector, Software Studies Initiative, Calit2 + UCSD Professor, Visual Arts Departmentmanovich.lev@gmail.com

You can capture this lecture using any media and share it.Follow our research: softwarestudies.com

Cultural Visualization techniques

cultural analytics research @Software Studies Initiative at Calit2/UCSD - key people:

Dr. Jeremy Douglass | Postdoctoral ResearcherTara Zepel | PhD student, Art HistorySunsern Cheamanunkul | PhD student, Computer ScienceSo Yamaoka | PhD student, Computer ScienceWilliam Huber | PhD student, Art History

for the expanded list of participants, see softwarestudies.com

Our research is made possible by the support from:

Center for Research in Computing and the Arts (CRCA)California Institute for Telecommunication and Information (Calit2)NEH Office of Digital HumanitiesNational Energy Research Scientific Computing Center Singapore Ministry of EducationBergen University (Norway)UCHRI

Software Studies Initiative Collaborators:Yuri Tsivian, Department of Art History, University of Chicago: cinemetrics.lv | film analysisAdele Eisenstein: Digital Formalism project (Department for Theatre, Film and Media Studies (TFM), Vienna University; the Austrian Film Museum; Interactive Media Systems Group, Vienna University of Technology) | film analysisNetherlands Institute for Sound and Vision | television and motion graphics analysisSan Diego Museum of Contemporary Art | mapping art for an exhibitionIsabel Galhano Rodrigues, University of Porto, Portugal | gesture analysisDavid Kirsh, Cognitive Science, UCSD | dance video analysisPh.D. students, Art History, UCSD | art history and visual cultureJim Hollan, Cognitive Science, UCSD | visualization | cultural analytics softwareFalko Kuester, Structural Engineering, UCSD + Calit2 | visual analytics | cultural analytics softwareYoav Freund, Computer Science and Engineering, UCSD | machine learning and machine vision | cultural analytics softwareKay O’Halloran, Multimodal Analysis Lab, National University of Singapore | Mapping Asian Cultures projectGiorgos Cheliotis: Communication and New Media, National University of Singapore | Mapping Asian Cultures projectMatthew Fuller: Goldsmiths College, University of London | software studies, game studies

cultural analytics = quantitative analysis and visualization of cultural data

goals of cultural analytics: - being able to better represent the complexity, diversity, variability, and uniqueness of cultural processes and artifacts - develop techniques to describe the dimensions of cultural artifacts and cultural processes which until now received little or no attention (such as gradual temporal change)- create much more inclusive cultural histories and analysis - ideally taking into account all available cultural objects created in particular cultural area and time period (“art history without names”) - democratize cultural research by creating open-source tools for cultural analysis and visualization - create interfaces for exploration of cultural data which operate across multiple scales - from details of structure of a particular individual cultural artifact/processes to massive cultural data sets/flows

cultural analytics - typical steps:

-1) description (i.e, “culture into data”): a) manual: annotation, tagging b) automatic: software analysis of media; capturing user activity our focus: easy-to-use techniques for automatic description of visual and interactive media

2) optional: statistical data analysis

3) data visualization (reduction, summarization) and data mapping (expansion, outlining, layering)our focus: new visualization + mapping techniques appropriate for interactive exploration of large sets of visual objects

4) interpretation (humanities), or explanation (science), or correlation (social science)

visualization of cultural data - visualization types:

1) visualization without doing additional annotation / automatic analysis

a) display all objects in a set together organized by exiting metadata (for instance, dates, artist names, etc.)

b) sample and re-order (for instance: montage, slice)

2) visualization after doing additional data analysis/annotation

c) visualization of newly generated metadata (graph)

d) display objects organized by metadata (image graph)

examples:d1) 2D sorted viewd2) 2D image graph - using single feature for each dimensiond3) 2D image graph - using combination of features (PCA, etc.)

examples of manual annotations and results of automatic analysis of visual data as tab delimited files (Excel spredsheets)

data exploration using currently available software used in science and business - limited because does not show images or video or 3D models or other media

demo: Mondrian software

our techniques:

“analytical browsing” - interfaces which combine media browsing and graphs to enable visual exploration of cultural data

“The aim pursued with visual exploration is to give an overview of the data and to allow users to interactively browse through different portions of the data. In this scenario users have no or only vague hypotheses about the data; their aim is to find some. In this sense, visual exploration can be understood as an undirected search for relevant information within the data. To support users in the search process, a high degree of interactivity must be a key feature of visual exploration techniques.”Christian Tominski, Event-Based Visualization for User-Centered Visual Analysis, PhD Thesis, Institute for Computer Science, Department of Computer Science and Electrical Engineering, University of Rostock, 2006.

“Exploration denotes an undirected search for interesting features in a data set.”Kreuseler, M., Nocke, T., and Schumann, H. A History Mechanism for Visual Data Mining. In Proceedings of the IEEE Symposium on information Visualization (infovis'04) - Volume 00 (October 10 - 12, 2004). INFOVIS. IEEE Computer Society, Washington, DC, 49-56. 2004.

source: www.infovis-wiki.net.

Cultural Analytics software running on HIPerSpace (May 2009)

Cultural Analytics software running on HIPerSpace (May 2009)

Cultural Analytics software running on HIPerSpace (May 2009)

our techniques:

“image graphs” - graphs which show the actual media objects (as opposed to only points)

a video of cultural analytics software tool running on HIPerSpace (available on Youtube)

bringing the same techniques to the desktop:

demo: mapping development of Piet Mondrian career with our imageJ tool

currently in development: interactive desktop software for image set exploration (Flash/Flex/Flare)

recent examples of our image graphs:

a complete set of all Time covers (4553 images, 1923-2008) - www.flickr.com/photos/culturevis/

Google logos arranged by type of design modification - www.flickr.com/photos/culturevis/

our techniques:

“data mapping” - highlighting the structure a cultural artifact while displaying this artifact (all or sampled)

traditional visualization - reduction, summarization, abstraction“data mapping” (or “supervisualization” ) - augmentation, expansion

text visualizations www.flickr.com/photos/culturevis/

SIGGRAPH 09 “media species” poster (www.flickr.com/photos/culturevis/

our techniques:

“sampling and re-ordering” (“re-mapping”)

demo - representing a shot structure of a feature film using imageJ

our techniques:

visualization of user activity using automatic software analysis of gameplay video

game play timelines (available on softwarestudies.com)

SIGGRAPH 09 games poster

in progress: analysis and visualization of a set of 48,000 comic book/manga/graphic novels titles = 1,2 million pages = appr. 10 million panels

data processing @ NERSC (National Energy Research Scientific Computing Center) using some of the fastest supercomputers in the world

all our open-source tools are available from Google code depository