I think I found the fix:
hMod = comm.GMSKModulator('BitInput', false, 'InitialPhaseOffset', 0, 'PulseLength',4, 'BandwidthTimeProduct', 0.3, 'SymbolPrehistory', 1, 'SamplesPerSymbol', 8, 'OutputDataType', 'double')
hmod_step = comm.RaisedCosineReceiveFilter
modsig=step(hMod,g(2:end));
For the executable to work the 2nd line had to be added.