How to check Hadoop and Matlab Integrated Properly or Not ???
3 views (last 30 days)
Show older comments
Hi
We integrated Matlab R2016b with Hadoop-2.7.2 ...but we are not sure it is working properly ....how we can check the program is running on cluster and each node is contributing in processing ...
1. It is taking more time for map-reduce (on cluster with 50 Nodes ) compare to matlab map-reduce with single computer..
2. Where to set Matlab Distributed Computer Server Properties ...like how many nodes are there, parallel pool etc .
3.How to see Matlab+Hadoop Cluster Configuration in Matlab Interface ??
Please provide me all detail in answer Thanks
1 Comment
lov kumar
on 2 Jun 2019
Please help me.
how to fix this error:
Error using mapreduce (line 124)
The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.
Error in bg1 (line 9)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
I am using this code:
setenv('HADOOP_HOME','C:/hadoop-2.8.0');
cluster = parallel.cluster.Hadoop;
mr = mapreducer(cluster);
ds = datastore('hdfs://localhost:9000/lov/airlinesmall.csv','TreatAsMissing','NA',...
'SelectedVariableNames','ArrDelay','ReadSize',1000);
preview(ds)
outputFolder = 'hdfs://localhost:9000/results/out1';
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
'OutputFolder',outputFolder)
Accepted Answer
Rick Amos
on 12 May 2017
To look at whether MATLAB is running on the Hadoop cluster correctly, your best bet is to look at the Hadoop/Yarn Web UI. By default, this is:
http://hadoophostname:8088/
Where hadoophostname should be replaced by the hostname of the head node of Hadoop. During a mapreduce operation in MATLAB, you should see a running job in the web UI.
If you don't see a job running, it might be possible that the Hadoop installation you provided to MATLAB is not configured to run jobs in cluster mode. This can happen if the Hadoop property mapreduce.jobtracker.address found in ${HADOOP_INSTALL}/etc/hadoop/mapred-site.xml has not been set or has been set to "local". This property should be set to the hostname of the headnode of the cluster.
In a Hadoop cluster, the number of workers that are launched are controlled by Hadoop. By default, it will run as many workers as it can fit in the memory given to it.
More Answers (0)
See Also
Categories
Find more on Tall Arrays and mapreduce in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!