Matlab + Hadoop Integration ??

11 views (last 30 days)
Pulkesh  Haran
Pulkesh Haran on 7 May 2017
Answered: lov kumar on 2 Jun 2019

We Followed standard doc for integrating Hadoop-2.7.2 with Matlab-R2016b. * Link for document* : http://in.mathworks.com/help/mdce/configure-a-hadoop-cluster.html

We Completed up to this stage but we are confuse with the rest part....

The requirements are:

  1. MATLAB® Distributed Computing Server™ must be installed or available on the cluster nodes. See Install Products and Choose Cluster Configuration.
  2. If the cluster is running in Kerberos authentication that requires the Java Cryptography Extension, you must download and install the Oracle version of this extension to each MATLAB Distributed Computing Server installation. You must also perform this step for the MATLAB client installation. To install the extension, place the Java Cryptography Extension jar files into the folder ${MATLABROOT}/sys/jre/${ARCH}/jre/lib/security.
  3. You must have a Hadoop installation on the MATLAB client machine, that can submit normal (non-MATLAB) jobs to the cluster.
  4. The cluster must identify its user home directory as a valid location that the nodes can access. You must choose a local filesystem path and typically use a local folder such as /tmp/hduserhome or /home/${USER}. Set yarn.nodemanager.user-home-dir for Hadoop version 2.X.
  5. There is one Hadoop property that must not be "final." (If properties are "final", they are locked to a fixed predefined value, and jobs cannot alter them.)
  6. The software needs to append a value to this property so that task processes are able to correctly run MATLAB. This property is passed as part of the job metadata given to Hadoop during job submission.

This property is mapred.child.env, which controls environment variables for the job's task processes.

You must provide necessary information to the parallel.cluster.Hadoop object in the MATLAB client session. For example, see Run mapreduce on a Hadoop Cluster (Parallel Computing Toolbox) and Use Tall Arrays on a Spark Enabled Hadoop Cluster (Parallel Computing Toolbox).

*MAIN QUESTION IS .....

1. We are not able to see any Cluster Configuration in Home->Parallel->Manage_Cluster_Profile ???

2.what is Role of MJS in this and how to configure it.

3.we are not understanding what we have to export and import for rest of the worker (slave )and master node* ??

Thanks

  2 Comments
Walter Roberson
Walter Roberson on 7 May 2017
Which MATLAB release are you using?
For me, in R2017a, look in the MATLAB toolbar, in the column that has "Preferences" at the top and "Set Path" below that. The column is between the Layout and the Add-Ons columns. The third item in the column is Parallel; click on that and it is a menu that includes Manage Cluster Profiles.
Pulkesh  Haran
Pulkesh Haran on 10 May 2017
I am using R2016b Matlab and Hadoop-2.7.2. We did not finding any info there (on Path you Mention)..

Sign in to comment.

Accepted Answer

Kojiro Saito
Kojiro Saito on 8 May 2017
  1. When integrating with Hadoop, MATLAB does not use a cluster profile. So, it's not an issue that Hadoop cluster profile is not listed in "Manage Cluster Profiles".
  2. When integrating with Hadoop, MJS is not used. MATLAB uses Hadoop's job scheduler, so you don't need to configure in MATLAB side.
  3. For the rest of workers and nodes, I don't think you need to export and import.
If you have any question, please ask us.
  13 Comments
Pulkesh  Haran
Pulkesh Haran on 29 May 2017
In datanode logs... No error is showing.. All container are clear... Only .matlaberror folder is created with list of map task failed
Kojiro Saito
Kojiro Saito on 2 Jun 2017
All right. Is there any corrupted images in 50K files?
If not, you should contact to MathWorks Technical Support.

Sign in to comment.

More Answers (1)

lov kumar
lov kumar on 2 Jun 2019
How to fix this error. Please help
Error using mapreduce (line 124)
The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.

Categories

Find more on Cluster Configuration in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!