Code covered by the BSD License  

Highlights from
urlread2

5.0

5.0 | 4 ratings Rate this file 103 Downloads (last 30 days) File Size: 9.5 KB File ID: #35693

urlread2

by

 

17 Mar 2012 (Updated )

Generalizes HTTP requests, providing more control and access to input and output

| Watch this File

File Information
Description

Version 1.1

Replacement for urlread. Example functions in the urlread2 file show how to make equivalent calls as are made in urlread. Helper functions can be written (some provided) to provide additional functionality without always needing to modify the urlread2 code.

Specific improvements:
- improved unicode support
- improved binary retrieval support
- request and response header access
- response status access

A technical description of the implementation can be found at:
http://undocumentedmatlab.com/blog/expanding-urlreads-capabilities/

Acknowledgements

Rewrites Of Urlread And Urlwrite and New Useful Urlread Urlreadv inspired this file.

This file inspired Slack Matlab.

Required Products MATLAB
MATLAB release MATLAB 7.13 (R2011b)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (25)
14 Sep 2014 Maxwell Agnew

I'm trying to use urlread2 to make a post request to the Etrade API. Their documentation says:

"Since this is a POST request, the parameters are included in the request as XML or JSON"

Can urlread2 handle a 'POST' request with the xml? An example would be much appreciated.
Thanks!

22 Jul 2014 Francisco

Thanks

22 Jul 2014 Jim Hokanson

@Francisco,

My apologies on PATCH not working. This is a problem with the underlying Java classes. I'll try to look into alternative Java classes.

Jim

22 Jul 2014 Francisco

Hello.

http DELETE method works OK:

url ='https://api-fxpractice.oanda.com/v1/accounts';
header = http_createHeader('Authorization','Bearer XXXXXXXX-YYYYYYYYY');
urlread2(url,'DELETE','',header)

returns:

ans =

{
"id" : 619104742,
"instrument" : "EUR_GBP",
"units" : 1,
"side" : "buy",
"price" : 0.79643,
"time" : "2014-07-22T03:25:49.000000Z",
"type" : "BuyEntry"
}

Cheers

21 Jul 2014 Francisco

Thanks for your answer.

I get this error message when I use the PATCH method here:
header = http_createHeader('Authorization','XXXXXXXX-YYYYYYYYY');
params = {'units' '2'};
uparams = http_paramsToString(params);
url ='https://api-fxpractice.oanda.com/v1/accounts/1125870/orders/619104742';
urlread2(url,'PATCH',uparams,header)
Error using urlread2 (line 157)
Java exception occurred:
java.net.ProtocolException: Invalid HTTP method: PATCH

at java.net.HttpURLConnection.setRequestMethod(Unknown Source)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.setRequestMethod(Unknown Source)

What do you think is causing it?
Cheers

Francisco

21 Jul 2014 Jim Hokanson

@Francisco,

Yes, both are. I've exposed the entirety of the HTTP request and response so you can do anything you want with it.

Jim

20 Jul 2014 Francisco

Hello. Thanks for your Help. Are http PATCH and DELETE methods supported? Thanks

20 Jul 2014 Jim Hokanson

@Francisco,

My apologies on the confusing documentation. When providing headers you also need to provide the body so that the order of the inputs is maintained. In this case providing an empty body is fine.

urlread2 (url,'GET','',header)

One day I'd like to rewrite this code base so that this is unnecessary and clearer ...

Best of luck and let me know if you have any other questions.

Jim

19 Jul 2014 Francisco

Hello,

I'm a newbie in MATLAB, I want to use MATLAB send http GET, POST, PATCH and DELETE commands to a REST API:

https://api-fxpractice.oanda.com/v1/accounts

for identification a header has to be sent in this format:
Parameter:Authorization
Value:Bearer XXXXXXXX-YYYYYYYYY

So I type:
header = http_createHeader('Authorization','Bearer XXXXXXXX-YYYYYYYYY')

url = 'https://api-fxpractice.oanda.com/v1/accounts'

urlread2 (url,'GET',header)

and I get this message:
Error using urlread2 (line 180)
Function input: body, should be of class char, uint8, or int8, detected: struct

Is this a problem with the header?

Thanks

Francisco

13 Jul 2014 Jim Hokanson

@Dan,

Usually problems arise due to the site using javascript. To get around this I'll usually use a program called "Fiddler" (In Windows). I'll go to the site in my web browser and then look at Fiddler to see what requests are being made to the server. Look for a request and subsequent response from the site that contains the information that you want. Then look more closely at the request to see how you would make the same request.

Best of luck.

Jim

13 Jul 2014 Dan

Thought that URLREAD2 would alleviate the issues I have using URLREAD to retrieve the contents of a URL page.
y = urlread( 'http://www.realtor.com/international/listing-detail/Costa-Guimar%C3%A3es_DISTRITO-DE-BRAGA_PO_666733');

Would like to extract information from the page (e.g. price). While I can see all the info in the source page (in the browser), there is no info in the return value from URLREAD2. Is the server somehow trying to protect the info? Anything that could be done?

Thanks a lot.

25 Jun 2014 none poncho

Perfect solution for one of matlab's problems, but i'm experiencing difficulties in using it. I'm trying to upload binary contents. Can you please provide an example?

05 Mar 2014 Cary Belas

Perfectly solved my quesionts

28 Nov 2013 Clemens

This function solved my long waiting time of the java call findproxyforurl. Thank you!

21 May 2013 Alex

Very helpful, thanks! This gave me exactly what I needed.

23 Aug 2012 Julian

Thanks, I like it. I only wanted to see what my web-server was sending in a Header, and did the trick nicely. I am more comfortable with MATLAB than web scripting, so this was the easiest solution for me. With regard to my rating though, please bear in mind that I did not take it for a thorough test drive..

04 Apr 2012 Gerardo Manzo

Thanks for your availability. I'll wait for your example, maybe I can figure out better what you are suggesting.

03 Apr 2012 Jim Hokanson

Ah, that clarifies things a bit. That brings up a tricky issue. Your use case isn't necessarily envisioned by their server and you might be blocked temporarily. To minimize this possibility you can space out your queries in time, perhaps every 10 - 20 seconds. I've run into this case with Google Scholar where eventually they poppped up a Captcha to try and prevent me from making automatic queries. It is a bit frustrating that they don't provide a code interface which would minimize their server load while still allowing access to users.

It looks like you are making a GET request, which means that all of the parameters are attached to the URL during the request. I've described this briefly in:
http://undocumentedmatlab.com/blog/expanding-urlreads-capabilities/
Essentially the way to do what you are asking is to make an example request in your browser where you change the date ranges, see how those dates are placed into the request url. Then you learn to modify the request url appropriately. I have code which would help with this but it is not included in this package. I can work on including it. Basically the code first splits on '&' characters to get [property]=[value] pairs. Then you split on '=' to separate the property from the value. Then you want to take each property and value and decode them using the function urldecode. Once this is done it should be pretty obvious where your dates are in the request. You then just modify those dates, repackage all of the parameters, and add them onto the original url after the '?' symbol.

I'll try and post an example of this with new code by the end of the week.

02 Apr 2012 Gerardo Manzo

Sorry, I forgot to tell you the most important thing. I meant, the search on Google News. Now it should be clear.

02 Apr 2012 Jim Hokanson

When you mention different time intervals I assume you mean running the function multiple times over the course of many hours/days/weeks etc. If that is the case these links might be helpful:
http://www.mathworks.com/matlabcentral/answers/30481-how-to-automatically-run-a-matlab-function-at-a-particular-time-every-day
http://www.mathworks.com/support/solutions/en/data/1-361S45/index.html?product=ML&solution=1-361S45

01 Apr 2012 Gerardo Manzo

I used this function and I realized that in the output there's what I need, that is, the number of results of a search. Is it possibile to collect it automatically for different time intervals?

26 Mar 2012 Jim Hokanson

Doubtful, that error is coming because some router or Google server is blocking your ip address (like if you are using a computer in China). You would need to use a proxy to get around that. The fix I just uploaded may provide a more explicit error message than what it was previously giving you, but it should be just the same as if you entered that url in a browser. Note: with the fix the error message will be the output (instead of throwing an error), and the extras structure will have a status indicating that the error occurred and that the output is an error message.

26 Mar 2012 Shane Lin

Do you think if it will work if I add "urlConnection.setFollowRedirects(1);" to urlread? Thanks

25 Mar 2012 Jim Hokanson

Well that stinks. It works for me. There are two issues going on here.
1) The example should work, I'll find a better one!
2) I made some last minute changes to try and directly handle errors instead of letting them cause code errors further down in the code which were harder to debug. What I didn't realize was that the handler recognizes HTTP status codes and processes them as errors as well. In other words, I thought the 403 error code you are seeing would go through just fine and that you would need to check the status and the output to see that something was wrong.
I'll address these changes and upload a new version ASAP.

Thanks!
Jim

23 Mar 2012 Shane Lin

Just tested the get example:

??? Error using ==> urlread2 at 203
Java exception occurred:
java.io.IOException: Server returned HTTP response code: 403 for URL:
http://www.google.com/search?hl=en&query=Pittsburgh+weather

Updates
25 Mar 2012

Fixed GET example and code now returns HTTP status codes that are errors along with the error text instead of just throwing an error.

26 Mar 2012

Changed description

Contact us