Parallel pool shut down during the iteration

Hi everyone, I encountered the below errors during the iteration.

Please see the error info below.

Iteration 20 of 20 for XXXX completed.
IdleTimeout has been reached.
Parallel pool using the 'XXX' profile is shutting down.
Starting parallel pool (parpool) using the 'AAAA' profile ...
connected to XX workers.
Error using XXX...

Also in my home directory, I found some error files seem generated by JAVA.

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 262144 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=21819, tid=0x00007f9533511700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007f95498a1000):  JavaThread "WorkerSessionBuilder.initiate-lab0" daemon [_thread_in_vm, id=29582, stack(0x00007f9533411000,0x00007f9533512000)]

Stack: [0x00007f9533411000,0x00007f9533512000],  sp=0x00007f953350dd80,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xacb18a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4ff4db]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b

After setting up the JAVA by

export _JAVA_OPTIONS="-Xmx110g"

Those

hs_error_pidXXX.log files

disappeared, but it still crashes during the iteration.

Would you please have a look and give me some advice? I am very new to Matlab. Any help would be highly appreciated. Thanks in advance.

Answer 1

[Kojiro Saito]

Could you check hard/soft limits are sufficient both for maximum number of process (nproc) and maximum number of file open(nofile) in your Linux machine.

ulimit -a

will show current hard/soft limits. You might need to increase these values for large parallel execution, for example, from 1024 to 65536.

[Navid Amini]

Dear Kojiro, your solution is perfect for me. Thank you very much.