Debugging MEX file using gdb on Windows

I am trying to debug some mex files using gdb on windows (VS Code is not an option for external reasons). I have tried the following:

1.

gdb matlab
(gdb) run

This will launch MATLAB but then see the main thread exitting successfully before the GUI opens. This means that I am unable to view the stack, set breakpoints etc.

2.

gdb
(gdb) attach $PID
(gdb) cont

This will keep getting stuck on random SIGSEGVs and require running signal 0 in gdb to let the process continue. Can also get stuck in some operations indefinitely.

3.

gdb
(gdb) attach $PID
(gdb) handle SIGSEGV nostop
(gdb) cont

This will keep running through the segfaults in some of the threads (I posit that they are due to the java graphics libraries based on what I have seen). However this means that if my code segfaults, then I won't stop for it, which is annoying since knowing the state at that moment is very useful. More importantly, many functions will hang while the debugger is attached. For example uigetfile will not show the new window until the debugger is detached. This makes this setup useless to me.

My knowledge of gdb and inner workings of MATLAB is insufficient to understand how to get this to work, if it is even possible. Has anyone been able to get this to work? If so, how did you go about doing it?

[Torsten]

Answer from AI (without guarantee):

Debugging MATLAB and MEX files using GDB on Windows is notoriously challenging. Your observations are accurate: MATLAB relies heavily on Java for its GUI and uses internal SIGSEGV (and other memory exceptions) as part of its normal runtime flow (e.g., for its Just-In-Time compiler and Java memory management).

When GDB intercepts these signals or halts execution threads, it deadlocks the Windows messaging queue, causing UI elements like uigetfile to hang indefinitely.

Yes, it is possible to get this working without VS Code, but it requires a very specific initialization sequence and fine-tuned signal configuration in GDB.

Step-by-Step Solution

1. Launch MATLAB directly from GDB (Fixing the "Main Thread Exiting" issue)

On Windows, launching matlab.exe simply triggers a launcher process that spawns the actual engine and then terminates. This is why GDB tells you the thread exited successfully before the GUI even loads.

To bypass the launcher, you must target the actual underlying executable or run MATLAB in single-thread/no-window modes.

Option A: The -wait flag

Force the launcher process to block and remain alive:

bash

gdb --args "C:\Program Files\MATLAB\R202Xy\bin\win64\matlab.exe" -wait

Option B: No Desktop Mode (Highly Recommended for GDB)

If your MEX debugging doesn't strictly depend on the Java desktop GUI, bypass it entirely to eliminate the Java-related hangs and random signals:

bash

gdb --args "C:\Program Files\MATLAB\R202Xy\bin\win64\matlab.exe" -nodesktop -nosplash

2. Configure GDB to pass MATLAB's Internal Signals (Fixing the Hangs & Segfaults)

Instead of globally turning SIGSEGV to nostop (which renders your own code crashes uncatchable), you need to tell GDB to pass the exceptions directly to the program without printing/reporting them, but only during the boot sequence.

Once inside GDB, run these commands before typing run:

gdb

(gdb) handle SIGSEGV nostop noprint pass
(gdb) handle SIGILL nostop noprint pass
(gdb) set startup-with-shell off

• pass: Tells GDB that MATLAB expects this signal and knows how to handle it.
• set startup-with-shell off: Fixes a well-known Windows MinGW/GDB bug where spawned processes instantly hang or break connection hooks.

3. Catching Your Specific MEX Segfault

If you use handle SIGSEGV nostop, GDB won't break when your MEX file crashes. To get around this and isolate your code from MATLAB's noise, use conditional breakpoints or catchpoints.

Approach A: Break on entry, then re-enable SIGSEGV tracking

1. Start MATLAB via GDB with handle SIGSEGV nostop noprint pass.
2. In the MATLAB command window, run your MEX file once so its .mexw64 binary is loaded into memory.
3. Switch back to your GDB terminal and hit Ctrl+C to interrupt MATLAB.
4. Set a breakpoint at your MEX function's entry point:

gdb

(gdb) break mexFunction

5. Turn tracking back on specifically for your code's execution:

gdb

(gdb) handle SIGSEGV stop print
(gdb) continue

6. Trigger the MEX file again in MATLAB. GDB will now stop right at the entry point of your C/C++ code, and any subsequent SIGSEGV will successfully freeze the debugger exactly where your custom code faulted.

Approach B: Catching C++ Exceptions

If your MEX code uses C++ and throws unhandled exceptions that lead to terminates/faults, use a catchpoint instead of relying on the system SIGSEGV:

gdb

(gdb) catch throw

Summary Configuration Workflow

For the most stable workflow, create a .gdbinit file or type the following sequence:

bash

# 1. Start from command line
gdb --args "C:\Program Files\MATLAB\R202Xy\bin\win64\matlab.exe" -wait -nodesktop


# 2. Inside GDB config
(gdb) set startup-with-shell off
(gdb) handle SIGSEGV nostop noprint pass
(gdb) run


# 3. Inside MATLAB
>> my_mex_file(args); % Loads symbols


# 4. Back in GDB (Ctrl+C)
(gdb) break mexFunction
(gdb) handle SIGSEGV stop print
(gdb) continue

Answer 1

[dpb]

Check out using VS Code instead of Visual Studio at <MEX Debugging. Redefined>.