Tech or Treat: The Sentimental Stock Saga

Proposal

Data Mining and Discovery

# GETTING THE LIBRARIES
if (!require(pacman))
  install.packages(pacman)

pacman::p_load(tidyverse,
               formattable,
               janitor,
               dlookr,
               here)

High-level goal

Sentiment analysis of diverse data sources to predict Apple Inc. stock price movement.

Abstract

Our aim is to conduct sentiment analysis on Apple Inc. stock by analysing diverse sources, including media coverage, industry reports, social media reviews, and investor opinions from news headlines. By leveraging sentiment analysis techniques, our project intends to uncover patterns in sentiment that may correlate with stock price movements. The insights gained could provide valuable perspectives on how public perception influences Apple’s stock performance, offering potential benefits for investors and analysts.

Dataset

# importing module to download data
import yfinance

# Using Ticker module in yfinace to get data of desired stock
apple = yfinance.Ticker("AAPL")
# usign history function to get stock data of yesterday with 1 hour interval from maket opening to closing
data = apple.history(period='1d',interval='1h')

The data is retrived from $Yahoo^\textregistered!\ Finance$ in real-time. This data contain’s the information about a particular stock and how it performed during certain period. This information includes openning,closing and other related features of the stock. The following are the attributes(freatures) of the data retrived:

Data Dictionary
Column Name	Data Type	Description
Datetime	String	This column represents the date and time for each data point, in the format “YYYY-MM-DD HH:MM:SS-TZ.” It provides the timestamp at which the stock price information was recorded. The timezone (TZ) indicates the time zone for the recorded data.
Open	Integer	This column contains the opening price of the stock at the specific timestamp. The opening price is the price at which the stock started trading at the beginning of the given time interval (e.g., each hour).
High	Integer	The “High” column contains the highest price reached by the stock during the time interval. It represents the peak value of the stock’s price within that hour.
Low	Integer	In this column, you find the lowest price reached by the stock during the same time interval. It represents the minimum value of the stock’s price within that hour.
Close	Integer	The “Close” column provides the closing price of the stock at the end of the specified time interval. It is the last price recorded before the end of that hour.
Volume	Integer	This column represents the trading volume of the stock during the given time interval. Trading volume is the total number of shares or contracts traded during that hour.
Dividends	Integer	The “Dividends” column typically contains information about any dividend payments made during the specified time interval. In the dataset you provided, it appears that no dividends were recorded (values are zero).
Stock Splits	Integer	The “Stock Splits” column would typically record any information related to stock splits that occurred during the specified time interval. In the dataset, it also appears that no stock splits were recorded (values are zero).

data

	Open	High	Low	Close	Volume	Dividends	Stock Splits
Datetime
2023-11-07 09:30:00-05:00	179.179993	180.820007	179.009995	180.710007	16268674	0.0	0.0
2023-11-07 10:30:00-05:00	180.695007	181.595001	180.529999	181.455002	8190077	0.0	0.0
2023-11-07 11:30:00-05:00	181.455002	181.639999	180.520004	181.175003	7493420	0.0	0.0
2023-11-07 12:30:00-05:00	181.175003	182.160004	180.934998	182.014999	6208182	0.0	0.0
2023-11-07 13:30:00-05:00	182.018494	182.440002	181.929993	182.340897	5219872	0.0	0.0
2023-11-07 14:30:00-05:00	182.350006	182.429993	181.899994	182.198303	5513486	0.0	0.0
2023-11-07 15:30:00-05:00	182.195007	182.339996	181.550003	181.820007	8142789	0.0	0.0

Headlines Dataset

import requests
import pandas as pd
from dateutil import parser
from datetime import datetime, timedelta

api_key='9dcab5d0d86940459623ec7dea5c8d36'
stock_symbol ="AAPL"
query_params = {
    'q': f'{stock_symbol}',
    'apiKey': api_key,
    'language': 'en',  # English language
    'country': 'us',   # USA sources
}

end_date = datetime.now()
start_date = end_date - timedelta(days=7)

from_date = start_date.strftime("%Y-%m-%d")
to_date = end_date.strftime("%Y-%m-%d")

news_url = f"https://newsapi.org/v2/everything?q={stock_symbol}&apiKey={api_key}&from={from_date}&to={to_date}&language=en"

response = requests.get(news_url)

if response.status_code == 200:
    news_data = response.json()
    articles = news_data['articles']
    headlines = [(article['title'], article['publishedAt']) for article in articles]
else:
    print("Failed to retrieve news data.")

apple_related_headlines=[]
for headline, _ in headlines:
    try:
        date = parser.parse(_)
        if 'Apple' in headline:
            apple_related_headlines.append((headline, date))
    except ValueError:
        pass

apple_related_headlines.sort(key = lambda x: x[1])
news_dataframe=pd.DataFrame(apple_related_headlines, columns=['Headline', 'Date'])
news_dataframe.head()

	Headline	Date
0	SwitchBot for iOSがApple Watchのコンプリケーションに対応。	2023-11-06 05:24:22+00:00
1	Apple's New Feature Detects Water In USB-C Por...	2023-11-06 08:20:06+00:00
2	Charlie Munger's Apple Confidence, Raskin Take...	2023-11-06 10:26:19+00:00
3	Apple iPhone-maker Foxconn sees solid holiday ...	2023-11-06 12:16:18+00:00
4	Apple MacBook Pro 14-inch review: Huge amounts...	2023-11-06 17:14:11+00:00

Data Dictionary
Column Name	Data Type	Description
HeadLine	String	A text field containing headlines related to Apple, ranging from news updates to reviews and opinions.
Date	String	A datetime field representing the date and time when the headline was published or reported. The timestamps are in Coordinated Universal Time (UTC).

Why did we choose this data?

This dataset is suitable for analyzing the stock price movement due to its time series format with open, high, low, and close prices, as well as volume, dividends, and stock splits. The high-frequency data, recorded at hourly intervals, is valuable for short-term trading. These features provide essential data for analysis and modeling of the stock price movement.

Variables Involved:

News Headlines: Collecting a dataset of news headlines relevant to the stock market. These headlines will serve as a variable to assess the impact on stock prices.
Stock Prices: Gathering historical stock price data for a selected set of stocks within a particular sector. These will be the stocks to study for correlations.
Sector Information: Identifying and categorizing stocks into their respective sectors for sector specific analysis.

Variables to be Created:

Correlation Coefficients: Calculating the correlation coefficients to quantify the relationships between Apple’s stock price movements and news headlines, and between stocks within the same sector.
Market Sentiment Index: Creating a sentiment index based on the tone of the news headlines (positive, negative, or neutral sentiment) to gauge how news sentiment affects Apple’s stock prices.

Question

How can sentiment analysis of news headlines contribute to understanding the stock price movements of Apple Inc., and what insights can be gained regarding the interconnectedness of external factors and their impact on Apple’s stock performance in the broader market context?

Motivation:

The question sparks curiosity about how news headlines might affect Apple’s stock prices. By delving into sentiment analysis, we aim to uncover patterns and insights in the relationship between public sentiment, specific news events, and Apple’s stock performance. This exploration could provide valuable knowledge for investors, analysts, and anyone keen on understanding the dynamics of stock price movements and market influences on Apple.

Analysis plan

Approach for Question

To address the question in a general sense, we would employ a combination of data analysis and predictive modeling. By collecting historical data on stock prices and news headlines, we can examine the historical relationship between news sentiment and market movements. This analysis might involve quantifying sentiment in news headlines, identifying patterns, and exploring correlations. Furthermore, predictive models can be developed to forecast potential future stock price changes based on these insights. The goal is to gain a better understanding of how news impacts stock prices and, if possible, use this knowledge to make informed predictions about future market behavior.

Plan Summary:

Data Collection:

Gather a diverse dataset of Apple-related news headlines, industry reports, and social media reviews during the specified timeframe.
Collect historical stock price data for Apple Inc. corresponding to the same period.

Data Preprocessing:

Clean and preprocess the text data, including handling any language-specific characters, removing duplicates, and ensuring consistency.
Preprocess the stock price data, ensuring alignment with the timeframes of the news dataset.

Sentiment Analysis:

Utilize natural language processing techniques to perform sentiment analysis on the collected news headlines.
Categorize sentiments as positive, negative, or neutral to quantify the overall sentiment trends.

Correlation Analysis:

Identify key news events and occurrences during the specified timeframe. Analyze how the sentiment trends derived from news headlines correlate with Apple’s stock price movements.
Explore potential correlations between news events related to other stocks and subsequent impacts on Apple’s stock.

Visualization and Interpretation:

Create visualizations to illustrate the sentiment trends and stock price movements over time.
Interpret the findings, identifying patterns, anomalies, and potential cause-effect relationships.

Statistical Analysis:

Conduct statistical tests to validate the significance of observed correlations and trends.
Evaluate the strength and direction of correlations between sentiment and stock price movements.

Discussion and Conclusions:

Summarize the key findings and their implications for understanding the influence of news sentiment on Apple’s stock prices.
Discuss any broader market influences on Apple’s stock performance. Consider limitations, potential biases, and areas for future research.

Weekly Plan of Attack

Week	Weekly Tasks	Persons in Charge	Backup
until November 8^th	Explore and finalize the dataset and the problem statements	Everyone	Everyone
-	Complete the proposal and assign some high-level tasks	Everyone	Everyone
November 9^th to 15^th	Getting to know about `yfinance` library and news headlines	Everyone	Everyone
-	Data cleaning and Data pre-processing	Eshaan	Aravind
-	Question specific exploration and data categorization	Likith	Sanjay
November 16^th to 22^nd	Performing `Sentiment analysis` for Q1	Sanjay	Likith
-	Performing `Sentiment analysis` and find `correlations` for Q1	Vamsi	Aravind
-	Exploring on how to integrate our analysis with real-time prices.	Aravind	Likith
November 23^rd to 29^th	Generating remaining parts of the plots for Q1	Eshaan	Sanjay
-	Improving the generated sentiment analysis model	Sanjay	Vamsi
-	Start integrating quarto and our model	Likith	Eshaan
November 30^th to December 6^th	Refining the code for code review with comments	Everyone	Everyone
-	Making a few changes on the model to test it based on historical data	Vamsi	Aravind
-	Continue with the integration of quarto and our models	Aravind	Eshaan
December 7^th to 13^th	Complete the quarto website with presentable data	Everyone	Everyone
-	Review the model and debugging	Everyone	Everyone
-	Write-up and presentation for the project	Everyone	Everyone

Repo Organization

The following are the folders involved in the Project repository.

‘data/’: Used for storing any necessary data files for the project, such as input files.
‘images/’: Used for storing image files used in the project.
‘_extra/’: Used to brainstorm our analysis which won’t impact our project workflow.
‘_freeze/’: This folder is used to store the generated files during the build process. These files represent the frozen state of the website at a specific point in time.
‘.github/’: Folder for storing github templates and workflow.

Note:

These are the planned approaches, and we intend to explore and solve the problem statement which we came up with. Parts of our approach might change in the final implementation.

--- title: "Tech or Treat: The Sentimental Stock Saga" subtitle: "Proposal" format: html: code-tools: true code-overflow: wrap code-line-numbers: true embed-resources: true editor: visual code-annotations: hover categories: - Data Mining and Discovery execute: warning: false --- ```{r load_pkgs, message=FALSE, output=FALSE} # GETTING THE LIBRARIES if (!require(pacman)) install.packages(pacman) pacman::p_load(tidyverse, formattable, janitor, dlookr, here) ``` ## High-level goal Sentiment analysis of diverse data sources to predict Apple Inc. stock price movement. ## Abstract Our aim is to conduct sentiment analysis on Apple Inc. stock by analysing diverse sources, including media coverage, industry reports, social media reviews, and investor opinions from news headlines. By leveraging sentiment analysis techniques, our project intends to uncover patterns in sentiment that may correlate with stock price movements. The insights gained could provide valuable perspectives on how public perception influences Apple's stock performance, offering potential benefits for investors and analysts. Dataset ```{r} #| label: load-dataset #| message: false ``` ```{=html} <div class="cell" data-execution_count="1"> <div class="sourceCode cell-code" id="cb1"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a># importing module to download data <a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>import yfinance <a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-4" aria-hidden="true" tabindex="-1"></a># Using Ticker module in yfinace to get data of desired stock <a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>apple = yfinance.Ticker("AAPL") <a href="#cb1-6" aria-hidden="true" tabindex="-1"></a># usign history function to get stock data of yesterday with 1 hour interval from maket opening to closing <a href="#cb1-7" aria-hidden="true" tabindex="-1"></a>data = apple.history(period='1d',interval='1h')</code><button title="Copy to Clipboard" class="code-copy-button"></button></pre></div> </div> ``` The data is retrived from $Yahoo^\textregistered!\ Finance$ in real-time. This data contain's the information about a particular stock and how it performed during certain period. This information includes openning,closing and other related features of the stock. The following are the attributes(freatures) of the data retrived: | Column Name | Data Type | Description | |--------------------------------|------------------|----------------------| | Datetime | String | This column represents the date and time for each data point, in the format "YYYY-MM-DD HH:MM:SS-TZ." It provides the timestamp at which the stock price information was recorded. The timezone (TZ) indicates the time zone for the recorded data. | | Open | Integer | This column contains the opening price of the stock at the specific timestamp. The opening price is the price at which the stock started trading at the beginning of the given time interval (e.g., each hour). | | High | Integer | The "High" column contains the highest price reached by the stock during the time interval. It represents the peak value of the stock's price within that hour. | | Low | Integer | In this column, you find the lowest price reached by the stock during the same time interval. It represents the minimum value of the stock's price within that hour. | | Close | Integer | The "Close" column provides the closing price of the stock at the end of the specified time interval. It is the last price recorded before the end of that hour. | | Volume | Integer | This column represents the trading volume of the stock during the given time interval. Trading volume is the total number of shares or contracts traded during that hour. | | Dividends | Integer | The "Dividends" column typically contains information about any dividend payments made during the specified time interval. In the dataset you provided, it appears that no dividends were recorded (values are zero). | | Stock Splits | Integer | The "Stock Splits" column would typically record any information related to stock splits that occurred during the specified time interval. In the dataset, it also appears that no stock splits were recorded (values are zero). | : Data Dictionary {tbl-colwidths="\[20,15,65\]"} ```{=html} <div class="cell" data-execution_count="2"> <div class="sourceCode cell-code" id="cb2"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>data</code><button title="Copy to Clipboard" class="code-copy-button"></button></pre></div> <div class="cell-output cell-output-display" data-execution_count="4"> <div> <table class="dataframe table table-sm table-striped small" data-quarto-postprocess="true" data-border="1"> <thead> <tr class="header"> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th">Open</th> <th data-quarto-table-cell-role="th">High</th> <th data-quarto-table-cell-role="th">Low</th> <th data-quarto-table-cell-role="th">Close</th> <th data-quarto-table-cell-role="th">Volume</th> <th data-quarto-table-cell-role="th">Dividends</th> <th data-quarto-table-cell-role="th">Stock Splits</th> </tr> <tr class="odd"> <th data-quarto-table-cell-role="th">Datetime</th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th"></th> </tr> </thead> <tbody> <tr class="odd"> <td data-quarto-table-cell-role="th">2023-11-07 09:30:00-05:00</td> <td>179.179993</td> <td>180.820007</td> <td>179.009995</td> <td>180.710007</td> <td>16268674</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="even"> <td data-quarto-table-cell-role="th">2023-11-07 10:30:00-05:00</td> <td>180.695007</td> <td>181.595001</td> <td>180.529999</td> <td>181.455002</td> <td>8190077</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="odd"> <td data-quarto-table-cell-role="th">2023-11-07 11:30:00-05:00</td> <td>181.455002</td> <td>181.639999</td> <td>180.520004</td> <td>181.175003</td> <td>7493420</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="even"> <td data-quarto-table-cell-role="th">2023-11-07 12:30:00-05:00</td> <td>181.175003</td> <td>182.160004</td> <td>180.934998</td> <td>182.014999</td> <td>6208182</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="odd"> <td data-quarto-table-cell-role="th">2023-11-07 13:30:00-05:00</td> <td>182.018494</td> <td>182.440002</td> <td>181.929993</td> <td>182.340897</td> <td>5219872</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="even"> <td data-quarto-table-cell-role="th">2023-11-07 14:30:00-05:00</td> <td>182.350006</td> <td>182.429993</td> <td>181.899994</td> <td>182.198303</td> <td>5513486</td> <td>0.0</td> <td>0.0</td> </tr> <tr class="odd"> <td data-quarto-table-cell-role="th">2023-11-07 15:30:00-05:00</td> <td>182.195007</td> <td>182.339996</td> <td>181.550003</td> <td>181.820007</td> <td>8142789</td> <td>0.0</td> <td>0.0</td> </tr> </tbody> </table> ``` ## Headlines Dataset ```{=html} <div class="cell" data-execution_count="1"> <div class="sourceCode cell-code" id="cb1"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>import requests <a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>import pandas as pd <a href="#cb1-3" aria-hidden="true" tabindex="-1"></a>from dateutil import parser <a href="#cb1-4" aria-hidden="true" tabindex="-1"></a>from datetime import datetime, timedelta <a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-6" aria-hidden="true" tabindex="-1"></a>api_key='9dcab5d0d86940459623ec7dea5c8d36' <a href="#cb1-7" aria-hidden="true" tabindex="-1"></a>stock_symbol ="AAPL" <a href="#cb1-8" aria-hidden="true" tabindex="-1"></a>query_params = { <a href="#cb1-9" aria-hidden="true" tabindex="-1"></a> 'q': f'{stock_symbol}', <a href="#cb1-10" aria-hidden="true" tabindex="-1"></a> 'apiKey': api_key, <a href="#cb1-11" aria-hidden="true" tabindex="-1"></a> 'language': 'en', # English language <a href="#cb1-12" aria-hidden="true" tabindex="-1"></a> 'country': 'us', # USA sources <a href="#cb1-13" aria-hidden="true" tabindex="-1"></a>} <a href="#cb1-14" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-15" aria-hidden="true" tabindex="-1"></a>end_date = datetime.now() <a href="#cb1-16" aria-hidden="true" tabindex="-1"></a>start_date = end_date - timedelta(days=7) <a href="#cb1-17" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-18" aria-hidden="true" tabindex="-1"></a>from_date = start_date.strftime("%Y-%m-%d") <a href="#cb1-19" aria-hidden="true" tabindex="-1"></a>to_date = end_date.strftime("%Y-%m-%d") <a href="#cb1-20" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-21" aria-hidden="true" tabindex="-1"></a>news_url = f"https://newsapi.org/v2/everything?q={stock_symbol}&apiKey={api_key}&from={from_date}&to={to_date}&language=en" <a href="#cb1-22" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-23" aria-hidden="true" tabindex="-1"></a>response = requests.get(news_url) <a href="#cb1-24" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-25" aria-hidden="true" tabindex="-1"></a>if response.status_code == 200: <a href="#cb1-26" aria-hidden="true" tabindex="-1"></a> news_data = response.json() <a href="#cb1-27" aria-hidden="true" tabindex="-1"></a> articles = news_data['articles'] <a href="#cb1-28" aria-hidden="true" tabindex="-1"></a> headlines = [(article['title'], article['publishedAt']) for article in articles] <a href="#cb1-29" aria-hidden="true" tabindex="-1"></a>else: <a href="#cb1-30" aria-hidden="true" tabindex="-1"></a> print("Failed to retrieve news data.") <a href="#cb1-31" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-32" aria-hidden="true" tabindex="-1"></a>apple_related_headlines=[] <a href="#cb1-33" aria-hidden="true" tabindex="-1"></a>for headline, _ in headlines: <a href="#cb1-34" aria-hidden="true" tabindex="-1"></a> try: <a href="#cb1-35" aria-hidden="true" tabindex="-1"></a> date = parser.parse(_) <a href="#cb1-36" aria-hidden="true" tabindex="-1"></a> if 'Apple' in headline: <a href="#cb1-37" aria-hidden="true" tabindex="-1"></a> apple_related_headlines.append((headline, date)) <a href="#cb1-38" aria-hidden="true" tabindex="-1"></a> except ValueError: <a href="#cb1-39" aria-hidden="true" tabindex="-1"></a> pass <a href="#cb1-40" aria-hidden="true" tabindex="-1"></a> <a href="#cb1-41" aria-hidden="true" tabindex="-1"></a>apple_related_headlines.sort(key = lambda x: x[1]) <a href="#cb1-42" aria-hidden="true" tabindex="-1"></a>news_dataframe=pd.DataFrame(apple_related_headlines, columns=['Headline', 'Date']) <a href="#cb1-43" aria-hidden="true" tabindex="-1"></a>news_dataframe.head()</code><button title="Copy to Clipboard" class="code-copy-button"></button></pre></div> <div class="cell-output cell-output-display" data-execution_count="1"> <div> <table class="dataframe table table-sm table-striped small" data-quarto-postprocess="true" data-border="1"> <thead> <tr class="header"> <th data-quarto-table-cell-role="th"></th> <th data-quarto-table-cell-role="th">Headline</th> <th data-quarto-table-cell-role="th">Date</th> </tr> </thead> <tbody> <tr class="odd"> <td data-quarto-table-cell-role="th">0</td> <td>SwitchBot for iOSがApple Watchのコンプリケーションに対応。</td> <td>2023-11-06 05:24:22+00:00</td> </tr> <tr class="even"> <td data-quarto-table-cell-role="th">1</td> <td>Apple's New Feature Detects Water In USB-C Por...</td> <td>2023-11-06 08:20:06+00:00</td> </tr> <tr class="odd"> <td data-quarto-table-cell-role="th">2</td> <td>Charlie Munger's Apple Confidence, Raskin Take...</td> <td>2023-11-06 10:26:19+00:00</td> </tr> <tr class="even"> <td data-quarto-table-cell-role="th">3</td> <td>Apple iPhone-maker Foxconn sees solid holiday ...</td> <td>2023-11-06 12:16:18+00:00</td> </tr> <tr class="odd"> <td data-quarto-table-cell-role="th">4</td> <td>Apple MacBook Pro 14-inch review: Huge amounts...</td> <td>2023-11-06 17:14:11+00:00</td> </tr> </tbody> </table> </div> </div> </div> ``` | Column Name | Data Type | Description | |--------------------------------|------------------|----------------------| | HeadLine | String | A text field containing headlines related to Apple, ranging from news updates to reviews and opinions. | | Date | String | A datetime field representing the date and time when the headline was published or reported. The timestamps are in Coordinated Universal Time (UTC). | : Data Dictionary {tbl-colwidths="\[20,15,65\]"} #### Why did we choose this data? This dataset is suitable for analyzing the stock price movement due to its time series format with open, high, low, and close prices, as well as volume, dividends, and stock splits. The high-frequency data, recorded at hourly intervals, is valuable for short-term trading. These features provide essential data for analysis and modeling of the stock price movement. #### Variables Involved: - News Headlines: Collecting a dataset of news headlines relevant to the stock market. These headlines will serve as a variable to assess the impact on stock prices. - Stock Prices: Gathering historical stock price data for a selected set of stocks within a particular sector. These will be the stocks to study for correlations. - Sector Information: Identifying and categorizing stocks into their respective sectors for sector specific analysis. #### Variables to be Created: - Correlation Coefficients: Calculating the correlation coefficients to quantify the relationships between Apple's stock price movements and news headlines, and between stocks within the same sector. - Market Sentiment Index: Creating a sentiment index based on the tone of the news headlines (positive, negative, or neutral sentiment) to gauge how news sentiment affects Apple's stock prices. ## Question How can sentiment analysis of news headlines contribute to understanding the stock price movements of Apple Inc., and what insights can be gained regarding the interconnectedness of external factors and their impact on Apple's stock performance in the broader market context? ### Motivation: The question sparks curiosity about how news headlines might affect Apple's stock prices. By delving into sentiment analysis, we aim to uncover patterns and insights in the relationship between public sentiment, specific news events, and Apple's stock performance. This exploration could provide valuable knowledge for investors, analysts, and anyone keen on understanding the dynamics of stock price movements and market influences on Apple. ## Analysis plan ### Approach for Question To address the question in a general sense, we would employ a combination of `data analysis` and `predictive modeling`. By collecting historical data on stock prices and news headlines, we can examine the historical relationship between news sentiment and market movements. This analysis might involve quantifying `sentiment` in news headlines, `identifying patterns`, and exploring `correlations`. Furthermore, predictive models can be developed to forecast potential future stock price changes based on these insights. The goal is to gain a better understanding of how news impacts stock prices and, if possible, use this knowledge to make informed predictions about future market behavior. ## Plan Summary: ### Data Collection: - Gather a diverse dataset of Apple-related news headlines, industry reports, and social media reviews during the specified timeframe. - Collect historical stock price data for Apple Inc. corresponding to the same period. ### Data Preprocessing: - Clean and preprocess the text data, including handling any language-specific characters, removing duplicates, and ensuring consistency. - Preprocess the stock price data, ensuring alignment with the timeframes of the news dataset. ### Sentiment Analysis: - Utilize natural language processing techniques to perform sentiment analysis on the collected news headlines. - Categorize sentiments as positive, negative, or neutral to quantify the overall sentiment trends. ### Correlation Analysis: - Identify key news events and occurrences during the specified timeframe. Analyze how the sentiment trends derived from news headlines correlate with Apple's stock price movements. - Explore potential correlations between news events related to other stocks and subsequent impacts on Apple's stock. ### Visualization and Interpretation: - Create visualizations to illustrate the sentiment trends and stock price movements over time. - Interpret the findings, identifying patterns, anomalies, and potential cause-effect relationships. ### Statistical Analysis: - Conduct statistical tests to validate the significance of observed correlations and trends. - Evaluate the strength and direction of correlations between sentiment and stock price movements. ### Discussion and Conclusions: - Summarize the key findings and their implications for understanding the influence of news sentiment on Apple's stock prices. - Discuss any broader market influences on Apple's stock performance. Consider limitations, potential biases, and areas for future research. ## Weekly Plan of Attack | Week | Weekly Tasks | Persons in Charge | Backup | |:----------------:|-------------------|:----------------:|:----------------:| | until November 8^th^ | Explore and finalize the dataset and the problem statements | Everyone | Everyone | | \- | Complete the proposal and assign some high-level tasks | Everyone | Everyone | | November 9^th^ to 15^th^ | Getting to know about `yfinance` library and news headlines | Everyone | Everyone | | \- | Data cleaning and Data pre-processing | Eshaan | Aravind | | \- | Question specific exploration and data categorization | Likith | Sanjay | | November 16^th^ to 22^nd^ | Performing `Sentiment analysis` for Q1 | Sanjay | Likith | | \- | Performing `Sentiment analysis` and find `correlations` for Q1 | Vamsi | Aravind | | \- | Exploring on how to integrate our analysis with real-time prices. | Aravind | Likith | | November 23^rd^ to 29^th^ | Generating remaining parts of the plots for Q1 | Eshaan | Sanjay | | \- | Improving the generated sentiment analysis model | Sanjay | Vamsi | | \- | Start integrating quarto and our model | Likith | Eshaan | | November 30^th^ to December 6^th^ | Refining the code for code review with comments | Everyone | Everyone | | \- | Making a few changes on the model to test it based on historical data | Vamsi | Aravind | | \- | Continue with the integration of quarto and our models | Aravind | Eshaan | | December 7^th^ to 13^th^ | Complete the quarto website with presentable data | Everyone | Everyone | | \- | Review the model and debugging | Everyone | Everyone | | \- | Write-up and presentation for the project | Everyone | Everyone | ## Repo Organization The following are the folders involved in the Project repository. - **'data/':** Used for storing any necessary data files for the project, such as input files. - **'images/':** Used for storing image files used in the project. - **'\_extra/':** Used to brainstorm our analysis which won't impact our project workflow. - **'\_freeze/':** This folder is used to store the generated files during the build process. These files represent the frozen state of the website at a specific point in time. - **'.github/':** Folder for storing github templates and workflow. ::: callout-note ## Note: These are the planned approaches, and we intend to explore and solve the problem statement which we came up with. Parts of our approach might change in the final implementation. :::