File Exchange

image thumbnail

Download Daily Data from Google and Yahoo! Finance

version 1.1 (84.4 KB) by

Provides functions for getting data from both data sources as well as helper utility functions

266 Downloads

Updated

View License

Editor's Note: This file was selected as MATLAB Central Pick of the Week

This archive contains functions for downloading daily stock price information from both Google and Yahoo! Finance as well as helpful utility functions. It furthermore contains a (basic but decent) example of a unit test harness created using MATLAB's functional unit test framework.

The utility functions allow you to convert the Google-formatted ticker symbols to the Yahoo! format and vice versa. Since Google does not provide adjusted closing prices, a helper function that uses dividend information to calculate them is automatically called.

Both Google and Yahoo! seem to allow the use of financial data for PERSONAL, INFORMATIONAL use ONLY and do not warrant against inaccuracies. These discrepancies are fairly common and are the main reason that this suite uses both Yahoo! AND Google data -- only if both sources agree on a piece of data should you begin to trust it.

For questions and discussion, go to www.quantsupport.com.

Comments and Ratings (36)

RV

RV (view profile)

the periods are the number of seconds that have passed from 1970 jan 1st. however daylight savings time is taking into account, therefore depending on the date there is a difference of 3600 seconds for the date numbers. as far as for 'history&crumb=&#&#&#&#' i do believe that this changes as well

Cost

Cost (view profile)

Hey guys, Yahoo! finance changed again the URL and now it seems really hard to automate.
For example:
'https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1463522400&period2=1495058400&interval=1d&events=history&crumb=KGGVODYW44C'
This is a query for Apple. I noticed that the period1 & period2 vary from Ticker to Ticker.... The code 'KGGVODYW44C' seems to be every time the same. So, the only problem should be to understand how Yahoo! calculate the periods...

Eddy Trinklein

Has the interface to yahoo changed? The URL doesn't seem to work.

Andrei

Andrei (view profile)

Or even better, as Constantino confirmed, just replace "url1 =" line in getYahooDailyData.m with:

url1 = 'https://chart.finance.yahoo.com/table.csv?s='

Andrei

Andrei (view profile)

That's what I changed in getYahooDailyData.m to make it work (I also left original commented code):

%% 2. Load Data in a loop
h = waitbar(0, 'Getting daily data from Yahoo!');
for iTicker = 1:length(tickers)
    try
        %str = urlread([url1 tickers{iTicker} url2]);
        str = webread([url1 tickers{iTicker} url2]);
    catch
        % Special behaviour if str cannot be found: this means that no
        % price info was returned. Error and say which asset is invalid:
        close(h)
        error('getYahooDailyData:invalidTicker', ...
            ['No data returned for ticker ''' tickers{iTicker} ...
            '''. Is this a valid symbol? Do you have an internet connection?'])
    end
    % c = textscan(str, '%s%f%f%f%f%f%f', 'HeaderLines', 1, 'Delimiter', ',');
    c = str;
    if isBeforeR2013b
        %ds = dataset(c{1}, c{2}, c{3}, c{4}, c{5}, c{6}, c{7}, 'VarNames',{'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'AdjClose'});
        ds = dataset(c{:,1}, c{:,2}, c{:,3}, c{:,4}, c{:,5}, c{:,6}, c{:,7}, 'VarNames',{'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'AdjClose'});
    else
        ds = table(c{:,1}, c{:,2}, c{:,3}, c{:,4}, c{:,5}, c{:,6}, c{:,7}, 'VariableNames',{'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'AdjClose'});
    end
    %ds.Date = datenum(ds.Date, 'yyyy-mm-dd');
    ds = flipud(ds);
    data.(genvarname(tickers{iTicker})) = ds;
    
    waitbar(iTicker/length(tickers), h);
end

close(h)

There was also a warning about genwarname - which it will be removed in subsequent release.

Cost

Cost (view profile)

Hey. Yahoo finance changed the url link. To fix the issue you can contact me on: costantino.ragno@unicam.it

Nan Chang

Great! Thanks for sharing!

Virgil Stokes

How to get historical FX exchange rates?

Mu Qiao

Louis

Louis (view profile)

Is there a way to download google trend data through your utility function or if not, would you know another option.
Thx
Louis

Hamza

Hamza (view profile)

Hi Michael,

How may I change de code in order to download the daily currency exchange rate ?

Best regards

Almir,
I'm not able to reproduce this issue at all.

The error message you're receiving indicates that some function is calling itself over and over again in an infinite, recursive loop.

The provided code doesn't ever call itself recursively, so I can only guess at what's going wrong: perhaps you un-commented the example when trying to copy it, and you forgot to re-comment it before running?

Almir

Almir (view profile)

Hi,
I have tried to run this script but cant get it to work. When running the example I get this error code:

getGoogleDailyData
Maximum recursion limit of 500 reached. Use set(0,'RecursionLimit',N) to change the limit. Be aware that exceeding your
available stack space can crash MATLAB and/or your computer.

Error in getGoogleDailyData

I have tried to set a higher recursionlimit but then my Matlab crash.

All help is appreciated.

Best regards

yan bing

if it support to download currency exchange rate, such as "EURSGD=X"?

Chris FUNG

Nice pieces

João Santos

This is beneficial for you if you plan to do any sort of algorithmic, high-frequency, or any sort of automated trading

https://www.youtube.com/watch?v=voPSG9qOmzA

Hi, John. Thanks for the kind words, and you're definitely not doing anything wrong.

I'm not seeing quite the same behavior that you are seeing: when I feed in a start date that's well before the IPO, Google returns a time series that's truncated to the IPO onwards-- that should be exactly what you're looking for. This might depend on which ticker you're trying to access, though-- I was looking at MSFT on NASDAQ. Let me know which ticker you were trying, and I'll see if I can reproduce it.

Going more broadly, it seems to be rather tricky to automatically grab the IPO date from the available Google information. I can see it on-screen when I type in a ticker symbol and zoom the historical data chart out to "All": the chart display either goes back to the IPO or some date in about 1978 (some cutoff point, I'm sure), whichever comes more recently.

I can't find that date anywhere in the HTML source code for that page, though. Feel free to look through the HTML source yourself (it's the variable "www1" in getGoogleDailyData.m), and if you find anything I'll be happy to add it in.

My only other idea would be to build some sort of logic around the data scraping (lines 89-99 of getGoogleDailyData) to check the returned data and to dynamically adjust the start date until it gives the desired results. That will be REALLY slow, though, and as I mentioned above, it doesn't seem to be needed for at least some tickers.

Anyways, let me know your thoughts and feedback, and we might be able to make some improvement tweaks!

John

John (view profile)

Hi Michael. Overall good code, but I've a problem dealing with the Google data code, and I wonder if you might help me with it, or tell me if you think I'm doing something wrong.

I'd like to be able to grab all of the historical data for a given stock, without assuming that I know the IPO date. This will allow me to cycle through many stocks and collect all of their data in an automated way.

However, I've run into a problem. The code which grabs Google historical data, in your suite, seems to require a start date for its data collection which is after the IPO date of the stock. If I feed it a start date before the IPO, the output reverts to give me only data from the last year or so. This seems to be a Google Finance interface issue... not necessarily just an issue with your code. The same thing happens when I try the other Google Finance data grabbers posted on Mathworks.

It seems that the only way to avoid this is to have prior knowledge of the IPO date for each stock. That might sound easy... until you consider that I'm trying to automate an application which grabs data for thousands of stocks, and I don't want to look each one up individually. If I have to, I'll figure out how to create an automated database of IPO dates... and then refer to it when I call your Google data grabber... but that's more work than I'd hoped and I'd rather avoid it if I can.

Do you have any thoughts?

Jan

Jan (view profile)

works very well.

Frédéric

Steven: The error message indicates that calculateAdjustedClose.m is not on the MATLAB path.

It is certainly included in the above ZIP-archive, so please ensure that you have extracted ALL of the files in the archive and added them to the path.

I'm running 2013a MATLAB with the Statistics Toolbox. I get an error when I run the example:
>> [data, dividends] = getGoogleDailyData({'NASDAQ:MSFT', 'EPA:ML'}, ...
   '01/01/2010', '01/01/2013', 'dd/mm/yyyy');

Undefined function 'calculateAdjustedClose' for input
arguments of type 'double'.

Error in getGoogleDailyData (line 128)
    ds.AdjClose = ...

What is the issue?

francesco

David

David (view profile)

Nevermind! Got it working. Just had to replace "today" with a lowercase "now"

David

David (view profile)

Any advice??
I try and execute
[stock_data] = get_gf_histdata('NYSE:GE');

and i get :
Undefined function or variable 'today'.

Error in get_gf_histdata (line 23)
enddate=[datestr(today,3) '+' datestr(today,7) '%2C+' datestr(today,10)];
 

Benjamin

Benjamin

Excellent Submission, I was using excel for this using Hoadley Option Add in... Scrap that now.

Amazingly simple to use file.

Thanks Michael!

Thanks for catching that, Hugh. I had forgotten that the TODAY function is a part of Financial Toolbox (for some strange reason). You can always replace TODAY with FLOOR(NOW), or (in the context that this function applies it) you can just use NOW.

I've posted an update to this File Exchange submission accordingly, and it should publicly appear soon. Thanks!

Hugh

Hugh (view profile)

The yahoo code worked for me, but the google code has an error: "today" is a valid matlab function.

I changed "today" to "date", but this leads to other errors.

Thanks.

francesco

mohammed

In your first point, I'm not certain what you mean. When DATEVEC is used in this code, it's always passed dates in a numeric format (which is a valid syntax). Date formatting conventions don't enter the picture as far as DATEVEC is concerned.

Hi, engineer!

In your first point, I'm not certain what you mean. When DATEVEC is used in this code, it's always passed dates in a numeric format (which is a valid syntax). Date formatting conventions don't enter the picture as far as DATEVEC is concerned.

For your second point, you're correct that the code does not make any adjustments for splits. In fact, only Google even reports when the splits occurred. My investigations concluded that both Yahoo! and Google already adjust for splits in both their raw closing prices and their volume data, so I just took that as a given. I'm happy to hear if you find that further adjustments need to be made.

Resampling the data to weekly or monthly intervals is beyond the scope of this submission. Perhaps you can create such an enhancement and submit it to the file exchange!

Finally, I'm unsure what you mean by your comment involving plotting: there is no data plotting in this submission. That said, I've found the dynamic date tick submission at http://www.mathworks.co.uk/matlabcentral/fileexchange/27075-intelligent-dynamic-date-ticks to be incredibly useful when I try to plot time series like these.

engineer

Pretty good Michael. There are two errors in your code, however:

1.
datevec() uses mm/dd/yyyy format (which is well established in the U.S.), not dd/mm/yyyy. This makes a difference in the data that is returned.

2.
Also, for splits, I believe that the volume must also be adjusted. I havn't looked into this yet, so I'm not sure how that will end up looking.

This code could be modified to also return weekly and monthly data (not just daily). This may require a separate query which you would then need to edit your existing daily structure (or Table) to add the OHLC weekly/monthly data. An alternative is to calculate these directly from the daily OHLC, however, non-trading days would make this rather cumbersome.

After returning the data from Yahoo!, datenum() is used so that the data can be easily plotted. Within the plot function, it is possible to get the handle for x-axis and plot the actual date corresponding to the matlab generated number.

Updates

1.1

Small change to getGoogleDailyData to remove dependence on Financial Toolbox.

MATLAB Release
MATLAB 8.2 (R2013b)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video