I have been attempting to web scrape data from yahoo finance specifically on historical data of S&P 500 which its webpage url is '/%5EGSPC/history/?period1=1574074965&period2=1731927744'

As you can see from the picture below, it seems that yahoo finance is not providing access to my act of web scraping. Is there any other solution to overcome this and web scrape the data

S&P 500 data

I have been attempting to web scrape data from yahoo finance specifically on historical data of S&P 500 which its webpage url is 'https://finance.yahoo/quote/%5EGSPC/history/?period1=1574074965&period2=1731927744'

As you can see from the picture below, it seems that yahoo finance is not providing access to my act of web scraping. Is there any other solution to overcome this and web scrape the data

S&P 500 data

• python
• web-scraping
• beautifulsoup

• Can you confirm whether you want the Table of content available in the URL you provided? – Moses01 Commented Nov 18, 2024 at 11:59
• Please make sure to post code and console output as code-fenced text, not images. Images can't be copy-pasted, load slower, take more bandwidth, and are less accessible. – Anerdw Commented Nov 19, 2024 at 0:21

1 Answer 1

Since the URL you use is Yahoo Finance and it is redirecting to multiple sites and fetching the data, but the beautifulsoup you use can try fetch up to 30 redirects only.

Instead of using Web Scrap, You can use Yahoo Finance module for Python. I noticed, You wanted to fetch the data from Nov 18,2019 to Nov 18,2024

So please use this code below to get the required data. You can change the dates as per your wish or use the below line to get all data

data = sp500.history(period="max")

Here is the code you should use:

import yfinance as yf
ticker = "^GSPC"
data = yf.Ticker(ticker)
hist = data.history(start="2019-11-18", end="2024-11-18")  # Specify date range
hist.to_csv("sp500_data.csv")