MATLAB Answers

1

How can i access .mat files, stored in an encrypted amazon S3 bucket via Matlab (API Key available)

Asked by David Reiter on 22 Mar 2019
Latest activity Commented on by David Reiter on 26 Mar 2019
Using Matlab R2018b on Windows 7
Already tried the following:
url = "https://se.eu-central-1.amazonaws.com/my_bucket/my_file.mat";
opt = weboptions('keyName','my_key_id','keyValue','my_api_key');
file = webread(url,opt);

  1 Comment

The exception i got:
Error using readContentFromWebService (line 62)
The server returned the status 403 with message "Forbidden" in response to the request to URL
https://s3.eu-central-1.amazonaws.com/bogie-diagnostic-978814483201/Parametrization/RRX/configWheelWear.mat.

Sign in to comment.

1 Answer

Answer by Rylan Dmello on 22 Mar 2019
 Accepted Answer

Hi David, it might be easier to use FileDatastore with the load function to do this:
% Set up S3 credentials.
setenv('AWS_ACCESS_KEY_ID', MY_ACCESS_KEY_ID);
setenv('AWS_SECRET_ACCESS_KEY', MY_SECRET_ACCESS_KEY);
setenv('AWS_REGION', MY_REGION);
% Construct a FileDatastore with a S3 URI.
fds = fileDatastore("s3://rdmello/*.mat", "ReadFcn", @load);
% Read from a single MAT file.
data = read(fds);
% Read all the data from all the MAT files on S3.
all_data = readall(fds);
Note here that MY_ACCESS_KEY_ID, MY_SECRET_ACCESS_KEY, and MY_REGION are strings that you need to provide in your code. See the Working with Remote Data doc page for more information about this.
Using a datastore here provides some benefits...each file on Amazon S3 is only downloaded only when needed, and you can build a tall array over it for easier scalability on parallel clusters.

  2 Comments

Hi Rylan,
thanks for your answer! I appretiate it.
Apperently I have trouble with my API-Key. When loading my .mat file via AWS-CLI I get an "access denied" error, which tells me that I still do not have necessary permissions. I tried loading the file with your code anyway. Matlab throws the following error:
Error using fileDatastore (line 105)
Cannot find files or folders matching: 's3-URI'.
I suppose I get this error due to my lack of permission? As soon as I get my new access key, I am going to comment on this page whether I had success or not.
Regards,
David
Hi again.
I got my new API-Key and guess what?! It worked just fine. Thanks again for your input. Here is the code that worked for me:
% Set up S3 credentials.
MY_ACCESS_KEY_ID = 'my_access_id';
MY_SECRET_ACCESS_KEY = 'my_access_key';
MY_REGION = 'my_region';
setenv('AWS_ACCESS_KEY_ID', MY_ACCESS_KEY_ID);
setenv('AWS_SECRET_ACCESS_KEY', MY_SECRET_ACCESS_KEY);
setenv('AWS_REGION', MY_REGION);
% Construct a FileDatastore with a S3 URI.
% file_key can be obtained directly on S3 when navigating to the deisred file or folder
fds = fileDatastore('s3://<my_bucket>/<file_key>.<file_extendsion>', "ReadFcn", @load,'FileExtensions','.mat');
% Read from a single MAT file.
data = read(fds);

Sign in to comment.