Advisor: Dr. Shang Rui Wu Master Thesis...
Transcript of Advisor: Dr. Shang Rui Wu Master Thesis...
![Page 1: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/1.jpg)
Predicting stock price using sentiment analysis combining Twitter, search engine
and investor intelligence dataMaster Thesis Defense
Rui WuAdvisor: Dr. Shang
![Page 2: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/2.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 3: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/3.jpg)
Motivation
Stock market is an integral part of global economy.
United States has a market capitalization of $18.668 trillion (2012).
It has a profound economic impact on the economy and everyday people.
The stock market crash of 1929 was a key factor in causing the great depression of the 1930s
![Page 4: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/4.jpg)
Demand
A good prediction model for stock market forecasting is always highly desirable and would of wider interest.
● Lots of studies and researches● Yield significant profit
![Page 5: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/5.jpg)
Social Media Power
Very early indicators can be extracted from online social media to predict changes in various economic and commercial indicators.
Twitter, which is now one of the most popular microblogging services, has been extensively used for real time sentiment tracking and public mood modeling.
![Page 6: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/6.jpg)
Goal
Building a stock price prediction system combining Twitter, Search Engine and Investor Intelligence data.
![Page 7: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/7.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 8: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/8.jpg)
1. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. J. Bollen (2009)● Extracted six dimensions of mood(tension,
depression, anger, vigor, fatigue, confusion) using an extended version of the Profile of Mood States
● Result: ○ 2008 President Selection match ‘Tension’ ○ Thanksgiving match ‘vigor’
![Page 9: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/9.jpg)
1. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. J. Bollen (2009)
![Page 10: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/10.jpg)
2. Predicting financial markets: Comparing survey, news, twitter and search engine data. J. Bollen (2010)● Survey a range of online data sets:
Twitter sentiment, news headlines, investor survey, Google search queries.
● Correlations○ Daily: DJIA - Trade Volume: 0.88○ Weekly: Twitter Volume - GIS: 0.61
![Page 11: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/11.jpg)
3. Analyzing stock market movements using twitter sentiment analysis. Rao, T., and Srivastava, S. (2012).● Twitter feature generation and correlations
up to 0.88 correlations, average value 0.5● EMMS Prediction
91% direction accuracy.
![Page 12: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/12.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 13: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/13.jpg)
![Page 14: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/14.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 15: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/15.jpg)
Methodology
● Data filtering and cleaning● Sentiment analysis● Feature generations● Machine learning algorithms● Predicting input
![Page 16: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/16.jpg)
Twitter Dataset
● Time range: 2009/7/31 - 2009/12/31● Size: 58.4GB, 400 million tweets● Format: timestamp, userId, content
T 2009-07-31 23:21:12
U 214325436
W I just got my new iPhone from Apple store.
● 20-30% of all public tweets
![Page 17: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/17.jpg)
Data Filtering and Cleaning
![Page 18: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/18.jpg)
Sentiment Analysis
● LingPipe from Alias-i● Algorithm: computational linguistics● Training set is from Internet Movie Database
10,000 comments with labels.● Classified in to 3 classes
positive, neutral and negative
![Page 19: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/19.jpg)
Feature Generations
● Twitter Sentiment features generation● Finance features generation● Search engine features generation● Investor intelligence features generation
![Page 20: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/20.jpg)
Twitter Sentiment Features● Mt-Positive: total number of positive tweets● Mt-Negative: total number of negative tweets● Bullishness Bt:
● Message Volume:
● Agreement among positive and negative At:
![Page 21: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/21.jpg)
Finance Features
● Yahoo Finance API - Historical Stock Price Data● Close, Trade Volume, Open, High and Low
● Return: ● Close: Ct = ln Close t● Trade Volume:
● Volatility:
![Page 22: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/22.jpg)
Example Feature set
![Page 23: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/23.jpg)
Search Engine Data
● Google Insights Search● Search volume data by given time range.● Categories of terms
○ Investment○ Finance
● Frequency value from 0 to 100.
![Page 24: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/24.jpg)
Google Search Query - Investment
![Page 25: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/25.jpg)
Investor Survey Data
● The American Association of Individual Investors
● Vote on S&P 500● bullish + neutral + bearish = 100%● spread = bullish - bearish● 8 week bullish average
![Page 26: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/26.jpg)
Example of weekly feature set
![Page 27: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/27.jpg)
Machine Learning Algorithms
● Decision Tree○ Decision Stump○ Bootstrap Aggregating
● Regression○ Linear Regression○ Gaussian Regression
● Neural Network○ Radial Basis Function Network○ Multilayer Perceptron
![Page 28: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/28.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 29: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/29.jpg)
Correlations
● Daily sentiment & finance● Weekly sentiment & finance● Weekly GIS & finance● Weekly AAII & finance● Time Lag analysis
![Page 30: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/30.jpg)
Daily feature set
![Page 31: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/31.jpg)
Daily sentiment correlation
![Page 32: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/32.jpg)
Weekly sentiment & GIS correlation
![Page 33: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/33.jpg)
Time Lag Analysis
Hypothesis:Twitter sentimentcan predict stock price of near future.
![Page 34: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/34.jpg)
Machine Learning Result - Daily
![Page 35: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/35.jpg)
ML Daily & Weekly Comparison
![Page 36: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/36.jpg)
ML GIS & AAII comparison
![Page 37: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/37.jpg)
Prediction Result - Amazon Weekly
![Page 38: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/38.jpg)
ML training 90%, test 10%
![Page 39: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/39.jpg)
![Page 40: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/40.jpg)
Function GetRealTimeTweetsestablishConnection();query(names);FOR each marketDay FOR each tweet IN queue tweet = queue.take(); parse(tweet); IF tweet#language eqauls 'EN' AND tweet#content CONTAINS word in opinionWords AND tweet#content NOT CONTAINS 'http' OR 'www.' WRITE tweet TO file(name); ENDEND
Streaming Data
![Page 41: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/41.jpg)
Function GetRealTimeTweetsestablishConnection();query(names);FOR each marketDay FOR each tweet IN queue tweet = queue.take(); parse(tweet); IF tweet#language eqauls 'EN' AND tweet#content CONTAINS word in opinionWords AND tweet#content NOT CONTAINS 'http' OR 'www.' WRITE tweet TO file(name); ENDEND
Streaming Data
![Page 42: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/42.jpg)
Overview
● Introduction● Related Work● System Architecture● Methodology● Result and Analysis● Summary and Future Work
![Page 43: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/43.jpg)
Summary
● 3 sets of feature sets are generated and results show its strong correlations with stock price movement.
● A prediction system is built consists of the model training component and the real time data collection component.
![Page 44: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/44.jpg)
Future Work
● Improve Twitter filter for tweets closely related to stock market.
● Train my own sentiment classifier with manually labeled dataset.
● Other algorithms
![Page 45: Advisor: Dr. Shang Rui Wu Master Thesis Defensedslsrv1.rnet.missouri.edu/~shangy/Thesis/RuiWu2014.pdf3. Analyzing stock market movements using twitter sentiment analysis. Rao, T.,](https://reader034.fdocuments.net/reader034/viewer/2022043017/5f3987da417d4e6ac00ffb0a/html5/thumbnails/45.jpg)
Question ?