WebVectors: a Toolkit for Building WebInterfaces for Vector Semantic Models

Andrey Kutuzov1

Elizaveta Kuzmenko2

University of Oslo1

National Research University Higher School of Economics (Moscow)2

April 7, 2016Analysis of Images, Social Networks, and Texts (AIST’16)

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.I Full-scale experimenting and playing with them demands software

development skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.I Full-scale experimenting and playing with them demands software

development skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.I Full-scale experimenting and playing with them demands software

development skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.

I Full-scale experimenting and playing with them demands softwaredevelopment skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.I Full-scale experimenting and playing with them demands software

development skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Introduction

WebVectors is a free and open-source toolkit to deploy web servicesfor vector semantic models (for example, word2vec-like).

I Vector semantic models (word embeddings) are widely used in textanalysis tasks.

I They are extensively studied for English, but not for otherlanguages (to lesser degree for Russian).

I Training predictive semantic models is computationally expensive.I Full-scale experimenting and playing with them demands software

development skills and a bit of linear algebra knowledge.

The goal of WebVectors is to provide access to vector semantic modelsto digital humanities and linguistics communities.

2

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;

I trained with artificial neural networks;I based on word co-occurrences in large training corpora;I represent words as dense lexical vectors (neural embeddings);I words occurring in similar contexts have similar vectors;I one can find nearest semantic associates of a given word by

calculating cosine similarity between vectors.

3

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;I trained with artificial neural networks;

I based on word co-occurrences in large training corpora;I represent words as dense lexical vectors (neural embeddings);I words occurring in similar contexts have similar vectors;I one can find nearest semantic associates of a given word by

calculating cosine similarity between vectors.

3

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;I trained with artificial neural networks;I based on word co-occurrences in large training corpora;

I represent words as dense lexical vectors (neural embeddings);I words occurring in similar contexts have similar vectors;I one can find nearest semantic associates of a given word by

calculating cosine similarity between vectors.

3

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;I trained with artificial neural networks;I based on word co-occurrences in large training corpora;I represent words as dense lexical vectors (neural embeddings);

I words occurring in similar contexts have similar vectors;I one can find nearest semantic associates of a given word by

calculating cosine similarity between vectors.

3

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;I trained with artificial neural networks;I based on word co-occurrences in large training corpora;I represent words as dense lexical vectors (neural embeddings);I words occurring in similar contexts have similar vectors;

I one can find nearest semantic associates of a given word bycalculating cosine similarity between vectors.

3

Predictive distributional semantic models

I introduced in [Bengio et al., 2003] and [Mikolov et al., 2013];I CBOW and SkipGram learning algorithms implemented in word2vec

tool;I trained with artificial neural networks;I based on word co-occurrences in large training corpora;I represent words as dense lexical vectors (neural embeddings);I words occurring in similar contexts have similar vectors;I one can find nearest semantic associates of a given word by

calculating cosine similarity between vectors.

3

Predictive distributional semantic models

Nearest semantic associatesаист:1. цапля 0.521522. журавль 0.474983. ворона 0.443374. лебедь 0.409975. ворон 0.387996. куропатка 0.38521(From a model trained on the Russian National Corpus and RussianWikipedia)

4

WebVectors

5

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ? (колесо)

I visualize word vectors and their geometrical relations.I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ? (колесо)

I visualize word vectors and their geometrical relations.I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ?

(колесо)I visualize word vectors and their geometrical relations.I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ? (колесо)

I visualize word vectors and their geometrical relations.I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ? (колесо)

I visualize word vectors and their geometrical relations.

I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

I find semantic associates of a query word:радость: веселие, ликование, печаль...

I calculate exact semantic similarity between pairs of words:цветок, роза: 0.59065

I apply simple algebraic operations to word vectors (addition,subtraction, finding average vector for a group of words):самолёт is to крыло as машина is to ? (колесо)

I visualize word vectors and their geometrical relations.I get the raw vector (array of real values) for the query word.

6

Functions available at WebVectors

WebVectors is more than a simple interface to trained vectormodels!

I Apply Part-of-Speech filters to the query word and resulting words.I Download results via API (in .csv or .json).I Query several models simultaneously.I Browse with your mobile devices.

7

Functions available at WebVectors

WebVectors is more than a simple interface to trained vectormodels!I Apply Part-of-Speech filters to the query word and resulting words.

I Download results via API (in .csv or .json).I Query several models simultaneously.I Browse with your mobile devices.

7

Functions available at WebVectors

WebVectors is more than a simple interface to trained vectormodels!I Apply Part-of-Speech filters to the query word and resulting words.I Download results via API (in .csv or .json).

I Query several models simultaneously.I Browse with your mobile devices.

7

Functions available at WebVectors

WebVectors is more than a simple interface to trained vectormodels!I Apply Part-of-Speech filters to the query word and resulting words.I Download results via API (in .csv or .json).I Query several models simultaneously.

I Browse with your mobile devices.

7

Functions available at WebVectors

WebVectors is more than a simple interface to trained vectormodels!I Apply Part-of-Speech filters to the query word and resulting words.I Download results via API (in .csv or .json).I Query several models simultaneously.I Browse with your mobile devices.

7

Visualizations

For single words – plots of corresponding n-dimensional vectors.

8

Visualizations

For multiple words – showing interdependence, projecting semanticrelationships into 2-dimensional space, using t-SNE.

9

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;I run Apache or other web server you use, to start user interface

listening to HTTP queries.I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;

I create your models and put them into the ‘models’ folder;I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;I run Apache or other web server you use, to start user interface

listening to HTTP queries.I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;

I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;I run Apache or other web server you use, to start user interface

listening to HTTP queries.I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;I customize configuration;

I run Python script to load models into memory and start daemonlistening to queries via sockets;

I run Apache or other web server you use, to start user interfacelistening to HTTP queries.

I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;

I run Apache or other web server you use, to start user interfacelistening to HTTP queries.

I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;I run Apache or other web server you use, to start user interface

listening to HTTP queries.

I enjoy!

10

How to install

Under the hood:I Python 2.7;I Flask;I Gensim.

I install project and its dependencies at your Linux server;I create your models and put them into the ‘models’ folder;I customize configuration;I run Python script to load models into memory and start daemon

listening to queries via sockets;I run Apache or other web server you use, to start user interface

listening to HTTP queries.I enjoy!

10

Explore and install

Live demos:I RusVectores (http://ling.go.mail.ru/dsm/en/) for Russian;I Semantic Vectors (http://ltr.uio.no/semvec/en/) for

Norwegian and English.

Source code:https://github.com/akutuzov/webvectors

11

References

Bengio, Y., Ducharme, R., and Vincent, P. (2003).A neural probabilistic language model.Journal of Machine Learning Research, 3:1137–1155.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.(2013).Distributed representations of words and phrases and theircompositionality.Advances in Neural Information Processing Systems 26.

12

Q&A

WebVectors: a Toolkit for Building Web Interfaces for Vector SemanticModels

Thank you for your attention!Questions are welcome.

Analysis of Images, Social Networks, and Texts (AIST’16)

12