Thread Subject: URLREAD and password protected url...

I want to download data from a URL but it prompts for a username and
password.


I have the username and password, of course, but I cannot figure out
how to use URLREAD for this site.


I am running the current MATLAB version on Windows 2000.

Michael Robbins wrote:
> I want to download data from a URL but it prompts for a username
> and
> password.


if your url is <http://www.company.com/private/>,
your login 'login' and your password 'password', did you try to use
the URL
 <http://login>:password@www.company.com/private/


Jeremie

Michael,


I do not have Matlab running here, so I can't test it.. but have you
tried to use urlread with method = 'post' and params = 'cell_array',
where cell_array = {username;password}?


Like this


urldoc = urlread('http://www.foo.com','post',{'mylogin';'mypasswd');


Hope it works.

>> I want to download data from a URL but it prompts for a
username
>> and password.
>
> if your url is <http://www.company.com/private/>,
> your login 'login' and your password 'password',
> did you try to use the URL
> <http://login>:password@www.company.com/private/


Thanks for the help. I did try that. My login has a "/" in it (ie.
abc/def), I don't know if that's the problem:


 <http://abc/def>:pwd@www.company.com/private/

Michael Robbins wrote:
> Thanks for the help. I did try that. My login has a "/" in it
> (ie.
> abc/def), I don't know if that's the problem:
>
> <http://abc/def>:pwd@www.company.com/private/


Yes, I think the '/' should be encoded using a '%xx' but I did a test
and the login:password@ solution does not seem to work with
urlread...
Sorry.

In article <eec2a4f.0@WebX.raydaftYaTP>, Jeremie <invalid@nospam.com>
wrote:

>Michael Robbins wrote:
>> I want to download data from a URL but it prompts for a username
>> and
>> password.
>
>
>if your url is <http://www.company.com/private/>,
>your login 'login' and your password 'password', did you try to use
>the URL
> <http://login>:password@www.company.com/private/


Shouldn't that be

  <http://login:password@www.company.com/private/> ?

Note the placement of the '>'. I just used this syntax to get a file
via ftp and it worked just fine.

As for the '/' in the password, it might work as is because maybe the
'/' won't terminate the password part of the URL (since the '@' should
do that) or maybe you can try escaping it with a backslash or the %
notation.

--
Doug Schwarz
Eastman Kodak Company
douglas.schwarz@kodak.com

Doug Schwarz <douglas.schwarz@kodak.com> writes:

> In article <eec2a4f.0@WebX.raydaftYaTP>, Jeremie <invalid@nospam.com>
> wrote:
>
>>Michael Robbins wrote:
>>> I want to download data from a URL but it prompts for a username
>>> and password.
>>
>>
>>if your url is <http://www.company.com/private/>,
>>your login 'login' and your password 'password', did you try to use
>>the URL
>> <http://login>:password@www.company.com/private/
>
>
> Shouldn't that be
>
> <http://login:password@www.company.com/private/> ?
>
> Note the placement of the '>'. I just used this syntax to get a file
> via ftp and it worked just fine.

The > was inserted courtesy of the perpetually broken URL handling of
the MATLAB Central newsreader.


--
Peter Boettcher <boettcher@ll.mit.edu>
MIT Lincoln Laboratory
MATLAB FAQ: http://www.mit.edu/~pwb/cssm/

> Shouldn't that be
<http://login:password@www.company.com/private/> ?
> Note the placement of the '>'. I just used this syntax to get a
> file
> via ftp and it worked just fine.
>
> As for the '/' in the password, it might work as is because maybe
> the
> '/' won't terminate the password part of the URL (since the '@'
> should
> do that) or maybe you can try escaping it with a backslash or the %
> notation.


Thanks, but it doesn't work for me.


Maybe it's the website, not the technique. I can access the website
fine via MS Internet Explorer.


>> [s,status]=urlread('<http://u1/u2:p@www.c.com/private/>')


s = ''
status = 0

In article <eec2a4f.6@WebX.raydaftYaTP>,
 "Michael Robbins" <michael.robbins@us.cibc.com> wrote:

>Maybe it's the website, not the technique. I can access the website
>fine via MS Internet Explorer.
>
>
>>> [s,status]=urlread('<http://u1/u2:p@www.c.com/private/>')
>
>
>s = ''
>status = 0


I don't think you want the <> characters in there. Try

  [s,status]=urlread('http://u1/u2:p@www.c.com/private/')

--
Doug Schwarz
Eastman Kodak Company
douglas.schwarz@kodak.com

Doug Schwarz wrote:
<michael.robbins@us.cibc.com> wrote:
[s,status]=urlread('<http://u1/u2:p@www.c.com/private/>')
>>
>>s = ''
>>status = 0
>
>
> I don't think you want the <> characters in there. Try
>[s,status]=urlread('http://u1/u2:p@www.c.com/private/')


Yes, there shouldn't be any <>, but as I said in another post,
this way of handling login/password doesn't work with urlread, it
still need to be implemented by TMW I guess...
And I do not see any other solution...


Jeremie
PS: sorry for the > in the URL of my first post, it has been added
by the Matlab Central Newsreader as stated by Peter.

> I don't think you want the <> characters in there. Try
>
> [s,status]=urlread('http://u1/u2:p@www.c.com/private/')


That was my original attempt; it didn't work.


When I do this


[s,status]=urlread('http://www.c.com')


it works fine, so I think my connection is fine.


I wish I could give you my password, but I must not.

In article <eec2a4f.8@WebX.raydaftYaTP>, Jeremie <invalid@nospam.com>
wrote:

>Doug Schwarz wrote:
><michael.robbins@us.cibc.com> wrote:
>[s,status]=urlread('<http://u1/u2:p@www.c.com/private/>')
>>>
>>>s = ''
>>>status = 0
>>
>>
>> I don't think you want the <> characters in there. Try
>>[s,status]=urlread('http://u1/u2:p@www.c.com/private/')
>
>
>Yes, there shouldn't be any <>, but as I said in another post,
>this way of handling login/password doesn't work with urlread, it
>still need to be implemented by TMW I guess...
>And I do not see any other solution...


But it does work (for me, at least). I was able to fetch a file (my
startup.m file located in my matlab directory) via ftp with

s = urlread('ftp://login:password@mymachine/matlab/startup.m');

--
Doug Schwarz
Eastman Kodak Company
douglas.schwarz@kodak.com

Doug Schwarz wrote:
> But it does work (for me, at least). I was able to fetch a file
> (my
> startup.m file located in my matlab directory) via ftp with
>
> s = urlread('ftp://login:password@mymachine/matlab/startup.m');


Well, and it does not work for me with HTTP, I obtain:
<TITLE>401 Authorization Required</TITLE>
</HEAD><BODY>
<H1>Authorization Required</H1>
This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.<P>


and I am sure of the argument passed to urlread because I copy-pasted
it from Matlab to wget and it works fine there...
I had a look to urlread code but I'm not a java expert. Maybe the
code should use the obect Authentificator from java.net:
 <http://java.sun.com/j2se/1.4.1/docs/api/java/net/Authenticator.html>


My two cents...

In article <eec2a4f.11@WebX.raydaftYaTP>, Jeremie <invalid@nospam.com>
wrote:

>Doug Schwarz wrote:
>> But it does work (for me, at least). I was able to fetch a file
>> (my
>> startup.m file located in my matlab directory) via ftp with
>>
>> s = urlread('ftp://login:password@mymachine/matlab/startup.m');
>
>
>Well, and it does not work for me with HTTP, I obtain:

[snip]

>and I am sure of the argument passed to urlread because I copy-pasted
>it from Matlab to wget and it works fine there...
>I had a look to urlread code but I'm not a java expert. Maybe the
>code should use the obect Authentificator from java.net:
> <http://java.sun.com/j2se/1.4.1/docs/api/java/net/Authenticator.html>
>
>
>My two cents...


Ok, I'll buy that. I don't think I have any http sites here behind our
firewall that require authorization so I can't try anything else. It
looks like it works ok for ftp, but not for http. I guess Michael is
out of luck.

--
Doug Schwarz
Eastman Kodak Company
douglas.schwarz@kodak.com

Thanks guys, I'll `escalate' this to TMW developers...


meanwhile I guess I'll stick with perl :(

Michael Robbins wrote:
> Thanks guys, I'll `escalate' this to TMW developers...
>
> meanwhile I guess I'll stick with perl :(


I found a solution in hacking urlread...
you need to add a setRequestProperty sending your login and password:
 <http://www.zdnet.com.au/builder/webdesign/scripting/story/0,2000040414,20268689,00.htm>


in urlread, just add after line 67
urlConnection = url.openConnection;
>>> urlConnection.setRequestProperty('Authorization','Basic
xxxxxxxxxxxxxxxxxx');


where xxxxxxxxx is a base64 version of the string login:password
I used <http://www.opinionatedgeek.com/DotNet/Tools/Base64Encode/>
to obtain a base64 version of my login:password
.
If you want to automatize the process, this string can be extract
from your url using
url = java.net.URL('http://aa:bb@www.c.com/')
url.getUserInfo
then there must be a java function to do the conversion or you can
use Peter Acklam code to do so:
 <http://home.online.no/~pjacklam/matlab/software/util/datautil/base64encode.m>


Hope this helps !

> I found a solution in hacking urlread...
> you need to add a setRequestProperty sending your login and
> password:


It worked! Thank you. Good job.

Hi,

I'm also struggling with urlread and password protected websites. What if the user name is an email adress? So the code looks like

urlread('http://user@name.com:pass@www.company.com')

with the following error

??? Error using ==> urlread at 74
Either this URL could not be parsed or the protocol is not supported.


Maybe I got it wrong, but doesn't terminate the @ the password? How can this be solved? Or is this an error of the website (no protocol support?)?

Thanks for any help!
Chris