Multimedia Analytics Jianping Fan Department of Computer Science University of North Carolina at Charlotte Presentation Outlines What is Multimedia Analytics in my mind? What multimedia analytics can do for Multimedia Computing? What we (UNCC team) have done so far? What is Multimedia Analytics? Multimedia analytics is science of multimedia computing facilitated by visual interface for interactive users inputs & assessments! 1. Multimedia Rept. 2. Computing Hypos 3. Decision Function 4. Computing Results Visual Analytics Semantic Gap Machine Side Human Side Assess & Inputs multimedia computing 1. Multimedia Rept. 2. Computing Hypos 3. Decision Function 4. Computing Results What is Multimedia Analytics? Multimedia Analytics a. Machine-Based Multimedia Computing; b. Visualization of Multimedia Data, Knowledge, Hypotheses, Decision Functions & Results; c. Human-Computer Interaction for assessing & changing hypotheses for multimedia computing Loop What Multimedia Analytics can do for Multimedia Computing? Multimedia Analytics Multimedia Computing Knowledge & Hypotheses Visualization User-System Interaction & Assessment Decision making Decision adjustment Hypothesis visualization Hypothesis assessment sampling & projection Hypothesis updating What Multimedia Analytics can do for Multimedia Computing? Hypothesis & Decision Function Visualization and Assessment Computing Results & Knowledge Visualization, Exploration and Assessment Leveraging the advantages of both human beings on creative thinking and computers on large memory and computing capacity. What we have done so far a. Interactive Similarity Function Assessment What we have done so far a. Interactive Similarity Function Assessment What we have done so far a. Interactive Similarity Function Assessment What we have done so far a. Interactive Similarity Function Assessment What we have done so far a. Interactive Similarity Function Assessment Inter-cluster visual correlation visualization Scalability Feature Quality Evaluation What we have done so far a. Interactive Similarity Function Assessment Similarity function is suitable for measuring visual similarity contexts? Similarity function combination (kernel combination) is good? HD projection can precisely preserve original visual similarity contexts? Feature quality is good? What we have done so far Misleading Effects: Data Uncertainty Junk Images What we have done so far Duplicates/Near-Duplicates Misleading Effects: Data Uncertainty What we have done so far Multiple text terms may share similar semantic meaning! One single text term may have multiple semantic meanings! Misleading Effects: Data Uncertainty What we have done so far Multi-Modal Information Association Misleading Effects: Data Uncertainty What we have done so far Misleading Effects: Data Uncertainty What we have done so far b. Interactive Decision Function Evaluation SVM Decision Boundary Visualization & Assessment What we have done so far b. Interactive Decision Function Evaluation Concept 1Concept 2 GMM Model Visualization What we have done so far b. Interactive Decision Function Evaluation Interactive User Involvements via labeling What we have done so far Updating kernel combinations (kernel weights); Updating projection optimization criteria to preserve similarity better! Updating decision function: margin between positive samples and negative samples! Updating hypotheses for data representation & similarity characterization! b. Interactive Decision Function Evaluation Enlarge the margin between two classes! Larger margin has good generalization property! What we have done so far c. Interactive Knowledge Exploration What we have done so far c. Interactive Knowledge Exploration What we have done so far c. Interactive Knowledge Exploration What we have done so far c. Interactive Knowledge Exploration What we have done so far d. Collaborative Multimedia Analytics People have strong motivations (self or social) to perform collaborate multimedia understanding! What we have done so far d. Collaborative Multimedia Analytics What we have done so far d. Collaborative Multimedia Analytics Learning task and training group organization; Human-computer communication on data and knowledge representation; Human-human communication on hypotheses and knowledge. More information is available at:Q & A!