Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word...

12
6/26/17 1 Unsupervised Learning Presentor: Yevgeny Shapiro What is supervised learning? Labels What is unsupervised learning? Labels What is unsupervised learning? • Looking for inner structure

Transcript of Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word...

Page 1: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

1

Unsupervised LearningPresentor:Yevgeny Shapiro

Whatissupervised learning?

Labels

Whatisunsupervised learning?

Labels

Whatisunsupervised learning?

• Lookingforinnerstructure

Page 2: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

2

• Tonsofdata,veryfewlabels

• Canimprovethestartingpointofsupervisedmodels

• Learnfeatureswithoutaspecifictaskforsemi-supervisedlearning

• Selfsupervision

• Closertohowthehumanbrainworks

Whyuseunsupervised learning? Classicmodels

Clustering

Autoencoders

RBMs

Self- supervision

Unsupervised LearninginNLP

•Lotsofdata

•HighlyStructured

•Highlysparse

לבןשהסתתר מאחורי העמוד הגבוה ברחוב היה הכלב

NLPtasks

•Lemmatization•Part-of-speechtagging•Translation•Grammarcorrection

• Languagemodels– Thecatis[?]• P(White| The catis)=0.1• P(while| Thecatis)=0

Dogs à dogDrove à Drive

Page 3: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

3

EfficientEstimationofWordRepresentationsinVectorSpace

TomasMikolov,KaiChen,GregCorrado,JeffreyDean

Whylearnvectorrepresentations?

• Traditionally,wordsarerepresentedaslabels• The[id443]cat[id23]is[id554]white[id444]

• Thisencodingisarbitrary

• “Dog”issimilarto“Dogs”

• Knowledgetransferbetweensimilarwords

Whatkindofsimilarity?

• Semanticregularities• Dog,Cat• Israel,Jerusalem• Mother,Father

• Syntacticregularities• work,works• Run,Running• Fast,Faster

Previousmethods

• Onehot• Verysparse• Noknowledgetransferbetweensimilarwords

• LSA– Latentsemanticanalysis• UsesSVDfordimensionalityreduction

Page 4: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

4

Previousmethods

• LDA– LatentDirichlet allocations• Bayesiantopicmodel• Veryexpensivetocompute

NNLM– neuralnetworklanguagemodel

• Npreviouswordswith1-of-Vcoding

• Firstlayer- linearembedding• Secondlayer– hiddenlayer• Outputlayer- Softmax

Problem- Computationalcomplexity

• Simplemodelstrainedwithlotsofdataarebetterthancomplexmodelstrainedwithlessdata

• Solution:Tryandreducethemodelcomplexity!• How?• Startbyanalyzingpreviousneuralnetmodels

Computationalcomplexity- NNLM

• Npreviouswordswith1-of-V coding• Firstlayer- linearembedding• Second layer– hidden layer• Output layer- Softmax

• Totalcomplexity• N– #previouswords• D– sizeofinputlayer• H– sizeofhiddenlayer• V– Vocabularysize

Hierarch ical So ftmax

BottleneckHuffman cod in g

Page 5: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

5

• Problem:very sparsevocabulary• Softmax

• Solution:Approximatestandardsoftmax• HierarchicalSoftmax

HierarchicalSoftmax Computationalcomplexity- RNNM

• Nolimitofpreviouswords• Onehiddenlayer

• Total complexity• Worddimensionality=H• H– sizeofhiddenlayer• V– Vocabularysize

Hierarch ical So ftmax

Bottleneck

Conclusions

• Mostofthecomplexityiscausedbythenon-linearhiddenunit• Toreducecomplexity– removethehiddenunit!

FirstModel– Continuous Bag-of-Words

• Similar toNNLM• Projectionisshared

• Total complexity• N– #inputwords• D– sizeofinputlayer• V– Vocabularysize

NNLM:

Page 6: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

6

SecondModel– Continuous Skip-gram

• Total complexity• C– maximumdistanceofthewords• D– sizeofinputlayer• V– Vocabularysize

• Samplingclosewordsmore

Training

• Adagrad – well suitedforsparsedata• Smalllearningrateforcommonparameters/words• Largelearningrateforrareparameters/words

Results

• Differentkindsofsimilarity• Computingpairs

• King– Man+Woman=Queen

Results

Page 7: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

7

Results ContextEncoders:FeatureLearningbyInpainting

DeepakPathak PhilippKrahenbuhlJeffDonahue TrevorDarrellAlexeiA.Efros

Motivation

• Thevisualdataisstructure,butinamorecomplexwaythantext.

• Humanscaneasilyunderstandthestructure.

Contextencoder

• Autoencoder –mightcauseordinarycompression

• Denoising autoencoder – learntodifferbetweensignalandnoise,nothingsemantic

Page 8: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

8

Previousworks

• UnsupervisedVisualRepresentationLearningbyContextPrediction–discriminativemodel

• Doersch, Carl, Abhinav Gupta, andAlexei A.Efros

Previousworks

• UnsupervisedLearningofVisualRepresentationsusingVideos

Xiaolong Wang,Abhinav GuptaRoboticsInstitute,CarnegieMellonUniversity

Previousworks

• SceneCompletionUsingMillionsofPhotographs

• Hays, James, and Alexei A. Efros

SystemArchitecture

• 1.Encoder– similartoAlexnet, trainedfromscratch• Input:227x227,output:6x6x256(pool5ofalexnet)

• 2.Connection– fullyconnectedlayer• Doesn’tconnectdifferentfeaturesforcomputationalreasons• Stride1convolutionisdonelater,

toconnecteddifferentchannels

• 3.Decoder– 5up-convolutions• Output– originaltargetsize

Page 9: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

9

LossFunction

• ReconstructionLoss• - Abinarymaskcorrespondingtodroppedregions• F(X)– encodedimage

LossFunction

• Adversarial loss• TolearnagenerativemodelG(=Finourcase)• WelearnadiscriminativemodelD– Loglikelihoodforrealsamples

• D=1 for real samples, andD=0 forgenerated samples

• Total lossfunction

Regionmasks

• Centralblock• Workswellforinpainting• Learnslowlevelboundaryfeatureswhichdon’tgeneralizewell

• Randomblock• Randomrectangleblocksareremoved• Upto¼oftheimage• ThemasksstillhavesharpboundarieswhichCNNfeatureslatchon

• Randomregion• RandommasksfromPASCALVOC2012,deformed• Upto¼oftheimage

Evaluation– SemanticInpainting

• Nonoiseintheadversarialsetting

• Thediscriminatorisnotconditionedonthecontext

• Nopoolinglayers

Page 10: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

10

Evaluation– SemanticInpainting Evaluation– SemanticInpainting

Evaluation– SemanticInpainting TheProblemwithPSNR

Page 11: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

11

Evaluation– Featurelearning

• Adversarial lossdidnotconvergewithAlexnet• Nearestneighborofthemissingregionusingcontextfeatures• Alexnet wastrainedasasupervisedclassificationproblem

Evaluation

• Detection:Fast– RCNNframework• Segmentation:FCNN

Thankyou!

RNNandLSTM

Page 12: Labels Unsupervised Learning - Semantic Scholar · 2017. 8. 3. · Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

6/26/17

12

CharacterLevelModel


Sal Corrado - Portfolio

Sal Corrado - Portfolio

Applying Unsupervised Learning - MathWorks · Applying Unsupervised Learning3 Unsupervised Learning Techniques As we saw in section 1, most unsupervised learning techniques are a

Applying Unsupervised Learning - MathWorks · Applying Unsupervised Learning3 Unsupervised Learning Techniques As we saw in section 1, most unsupervised learning techniques are a

Scalable Representation Learning for Heterogeneous Networkso Computational lens on big social ... In NIPS ’13, pp. 3111-31119. 4. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient

Scalable Representation Learning for Heterogeneous Networkso Computational lens on big social ... In NIPS ’13, pp. 3111-31119. 4. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient

Unsupervised Learning Learning Unsupervised...Unsupervised Learning and Data Mining Learning Mining Unsupervised Learning and Data Mining Unsupervised Data Clustering Supervised Learning

Unsupervised Learning Learning Unsupervised...Unsupervised Learning and Data Mining Learning Mining Unsupervised Learning and Data Mining Unsupervised Data Clustering Supervised Learning

Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey

Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey

Unsupervised Learning. Supervised learning vs. unsupervised learning.

Unsupervised Learning. Supervised learning vs. unsupervised learning.

Boletin Tecnico Corrado

Boletin Tecnico Corrado

Word2vec embeddings: CBOW and SkipgramWord2vec Mikolov, Chen, Corrado & Dean (2013): E cient Estimation of Word Representations in Vector Space Comes in two avors: Continuous Bag Of

Word2vec embeddings: CBOW and SkipgramWord2vec Mikolov, Chen, Corrado & Dean (2013): E cient Estimation of Word Representations in Vector Space Comes in two avors: Continuous Bag Of

word2vec Hendrik Heuer From theory to practicehen-drik.de/pub/Heuer - word2vec - From theory to...word2vec • by Mikolov, Sutskever, Chen, Corrado and Dean at Google • NAACL 2013

word2vec Hendrik Heuer From theory to practicehen-drik.de/pub/Heuer - word2vec - From theory to...word2vec • by Mikolov, Sutskever, Chen, Corrado and Dean at Google • NAACL 2013

Unsupervised Learning and Data Mining Unsupervised Learning

Unsupervised Learning and Data Mining Unsupervised Learning

Unsupervised Cross-Modal Alignment of Speech …people.csail.mit.edu/andyyuan/docs/nips-18.unsupervised...2. Learning Embeddings Text Embedding Space • Train Word2Vec [Mikolov et

Unsupervised Cross-Modal Alignment of Speech …people.csail.mit.edu/andyyuan/docs/nips-18.unsupervised...2. Learning Embeddings Text Embedding Space • Train Word2Vec [Mikolov et

Corrado agosto 2013

Corrado agosto 2013

CORRADO TINTERRI

CORRADO TINTERRI

Club Corrado Technicité - Toute la technique sur le corrado

Club Corrado Technicité - Toute la technique sur le corrado

Corrado 94

Corrado 94

Unsupervised Cross-Modal Alignment of Speech …people.csail.mit.edu/andyyuan/docs/nips-18.unsupervised...Word2vec [Mikolov et al., 2013] Language 1 Language 2 Do not need to be parallel!

Unsupervised Cross-Modal Alignment of Speech …people.csail.mit.edu/andyyuan/docs/nips-18.unsupervised...Word2vec [Mikolov et al., 2013] Language 1 Language 2 Do not need to be parallel!

Disability Manager - unige.it...Corrado Roncallo - Mariacarla Sbolci 17 Grazie per l’attenzione Corrado Roncallo - Mariacarla Sbolci 18 Title Disability Manager Author Corrado Roncallo

Disability Manager - unige.it...Corrado Roncallo - Mariacarla Sbolci 17 Grazie per l’attenzione Corrado Roncallo - Mariacarla Sbolci 18 Title Disability Manager Author Corrado Roncallo

Corrado Malanga - Evideón

Corrado Malanga - Evideón

Corrado Giacomelli

Corrado Giacomelli

Corrado rabaglio

Corrado rabaglio