NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER...
-
Upload
andra-hunt -
Category
Documents
-
view
219 -
download
2
Transcript of NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER...
![Page 1: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/1.jpg)
NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY
EDGAR SIMO-SERRA
SANJA FIDLER
FRANCESC MORENO-NOGUER
RAQUEL URTASUN
CVPR2015
![Page 2: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/2.jpg)
OUTLINE
Introduction
Fashion144k Dataset
Discovering Fashion from Weak Data
Experimental Evaluation
Conclusions
![Page 3: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/3.jpg)
INTRODUCTION• Goal
To learn and predict how fashionable a person looks on a photograph.
Suggest subtle improvements the user could make to improve her/his appeal.
![Page 4: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/4.jpg)
INTRODUCTION
Photograph’s settingUser type
The type of outfit
The fashionability score
![Page 5: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/5.jpg)
INTRODUCTION• Novel dataset and Conditional Random Field model that
can reason about different aspects of fashionability.
Fashion144k dataset
chictopia.com
144,169 posts
CRF model
User Setting
Outfit
Fashionability
![Page 6: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/6.jpg)
INTRODUCTION• Contribution
A novel heterogeneous dataset.
Provides a detailed analysis of the data.
Analyzes outfit trends through the last six years of posts spanned by our dataset.
![Page 7: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/7.jpg)
FASHION144K DATASET• Data from crawling the largest fashion social site:
chictopia.com
• The dataset consists of 144,169 user posts.
• Normalized votes used as a proxy for fashionability.
• From March 2nd in 2008(the first post to the chictopia website) to May 22nd 2014.
![Page 8: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/8.jpg)
FASHION144K DATASET
Post Details-Votes-Comments-Favourites
Garments
Tags
Colours
Date Photo
User Details-Followers-Locations
Comments
![Page 9: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/9.jpg)
FASHION144K DATASET• Dataset Statistics
![Page 10: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/10.jpg)
FASHION144K DATASET• Influence of Compatriots
The mean score on a scale of 1 to 5 of the sentiment analysis [25] .
[25] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
![Page 11: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/11.jpg)
FASHION144K DATASET• Measuring Fashionability of a Post.
Power-law distribution
Try to eliminate the temporal dependency
- calculating histograms of the votes for each month
- fit a Gaussian distribution to it
We then bin the distribution such that the expected number of posts for each bin is the same.
- obtain a quasi-equal distribution of classes
- ranging from 1 (not fashionable) to 10 (very fashionable)
logarithm
![Page 12: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/12.jpg)
FASHION144K DATASET• The number of posts and fashionability scores mapped to the
globe
![Page 13: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/13.jpg)
FASHION144K DATASET• Fashionability of cities with at least 1000 posts
![Page 14: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/14.jpg)
DISCOVERING FASHION FROM WEAK DATA
• We make use of a Conditional Random Field (CRF) to learn the different outfits, types of people and settings.
• Our potentials make use of deep networks over a wide variety of features to produce accurate predictions of how fashionable a post is.
![Page 15: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/15.jpg)
DISCOVERING FASHION FROM WEAK DATA• More formally
- : a random variable capturing the type of user
- : the type of outfit
- : the setting
- : the fashionability of a post
• Represents the energy of the CRF variables
non-parametric pairwise potentials
![Page 16: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/16.jpg)
DISCOVERING FASHION FROM WEAK DATA• Overview of the different features used
![Page 17: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/17.jpg)
DISCOVERING FASHION FROM WEAK DATA• Deep network for feature extraction
Train four feature extractors jointly maximizing fashionability
Afterwards, the four networks are used independently
![Page 18: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/18.jpg)
DISCOVERING FASHION FROM WEAK DATA• Modelling fashion with a Conditional Random Field
User Setting
OutfitFashionability
![Page 19: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/19.jpg)
DISCOVERING FASHION FROM WEAK DATA• User
Fans : number of fans that the particular user has
Face detector Rekognition
Our user unary potentials
https://rekognition.com
![Page 20: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/20.jpg)
DISCOVERING FASHION FROM WEAK DATA• Outfit
Garments : bag-of-words with garment tags
Colours : bag-of-words with colour tags
Singles : bag-of-words with split colour tags (e.g., red-dress)
Our outfit unary potentials
![Page 21: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/21.jpg)
DISCOVERING FASHION FROM WEAK DATA• Setting
Scene classifier SUN Dataset [30]
Obtain features Caffe [12]
Location : distance from location clusters [24]
Our setting unary potentials
[30] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.[12] Y. Jia. Caffe: An open source convolutional architecture for fast feature embedding. http://caffe. berkeleyvision.org/, 2013.[24] E. Simo-Serra, C. Torras, and F. Moreno-Noguer. Geodesic Finite Mixture Models. In BMVC, 2014.]
![Page 22: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/22.jpg)
DISCOVERING FASHION FROM WEAK DATA• Fashion
∆T : time between post creation and download
Tags : bag-of-words with post tags
Comments : sentiment analysis [25] of comments
Style : style of the photography [13]
Our fashion unary potentials
[25] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. [13] S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agarwala, and T. Darrell. Recognizing image style. In BMVC, 2014.
![Page 23: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/23.jpg)
DISCOVERING FASHION FROM WEAK DATA• Correlations
We use a non-parametric function for each pairwise and let the CRF learn the correlations.
- Similarly for the other pairwise potentials.
![Page 24: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/24.jpg)
(1)-LEARNING AND INFERENCE• Estimate the initial latent states using clustering.
• Learn the CRF model using the primal-dual method of [9].
• We use the implementation of [22] and perform inference using message passing [21].
[9] T. Hazan and R. Urtasun. A primal-dual message-passing algorithm for approximated large scale structured prediction. In NIPS, 2010. [22] A. G. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Ef- ficient structured prediction with latent variables for general graphical models. In ICML, 2012.[21] A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Distributed message passing for large scale graphical models. In CVPR, 2011.
![Page 25: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/25.jpg)
EXPERIMENTAL EVALUATION(1)-CORRELATIONS
• Fashionability and Economy
• Fashionability and Face-related
1 ─ 7Most developed
Rich country
Least developed
Poor country
| 0.02 |↑
![Page 26: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/26.jpg)
EXPERIMENTAL EVALUATION(1)-CORRELATIONS• The mean estimated beauty and dominant inferred ethnicity on
the world map
![Page 27: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/27.jpg)
EXPERIMENTAL EVALUATION(2)-PREDICTING FASHIONABILITY• We use 60% of the dataset as a train set, 10% as a validation,
and 30% as test, and evaluate our model for the fashionability prediction task.
![Page 28: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/28.jpg)
EXPERIMENTAL EVALUATION(2)-PREDICTING FASHIONABILITY• Examples of true and false positives for the fashionability
classification task obtained with our CRF model.
![Page 29: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/29.jpg)
EXPERIMENTAL EVALUATION(2)-PREDICTING FASHIONABILITY• Analyze the individual contribution of each of the features
![Page 30: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/30.jpg)
EXPERIMENTAL EVALUATION(3)-IDENTIFYING LATENT STATES• Visualization of the dominant latent clusters for the settings
and outfit nodes in our CRF by country.
![Page 31: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/31.jpg)
EXPERIMENTAL EVALUATION(3)-IDENTIFYING LATENT STATES• Visualizing pairwise potentials between nodes in the CRF
![Page 32: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/32.jpg)
EXPERIMENTAL EVALUATION(4)-OUTFIT RECOMMENDATION• Estimate setting and user state and find outfit state that
maximizes fashionability.
Example of recommendations provided by our model
![Page 33: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/33.jpg)
EXPERIMENTAL EVALUATION(5)-ESTIMATION FASHION TRENDS• Visualizing outfit trends in Manila and Los Angeles
![Page 34: NEUROAESTHETICS IN FASHION: MODELING THE PERCEPTION OF FASHIONABILITY EDGAR SIMO-SERRA SANJA FIDLER FRANCESC MORENO-NOGUER RAQUEL URTASUN CVPR2015.](https://reader035.fdocuments.net/reader035/viewer/2022062804/5697bf901a28abf838c8e049/html5/thumbnails/34.jpg)
CONCLUSIONS• We presented a novel task of predicting fashionability of
users photographs.
• We collected a large-scale dataset by crawling a social website.
• Our model can be used to analyze fashion trends in the world or individual cities, and can also be used for outfit recommendation.