There are many ways to approach it, but if logical masking suffices, simple indexing works:
fg = imread('image.png');
bg = imread('peppers.png');
mask = repmat(rgb2gray(fg)>5,[1 1 3]);
roirows = offset(1):offset(1)+size(fg,1)-1;
roicols = offset(2):offset(2)+size(fg,2)-1;
roi = bg(roirows,roicols,:);
out(roirows,roicols,:) = roi;
If it's desirable to adjust FG opacity, that can also be done by multiplication. The ROI is essentially the weighted average of the FG and BG in the region defined by the logical mask.
fg = imread('image.png');
bg = imread('peppers.png');
mask = repmat(rgb2gray(fg)>5,[1 1 3]);
roirows = offset(1):offset(1)+size(fg,1)-1;
roicols = offset(2):offset(2)+size(fg,2)-1;
roi = bg(roirows,roicols,:);
roi(mask) = fgopacity*fg(mask) + (1-fgopacity)*roi(mask);
out(roirows,roicols,:) = roi;
Multiplicative composition is more expensive, but it is generally more flexible. Often, logical masks produce visually poor results due to the hard edges. It may be desirable to have a feathered mask. The opacity parameter can even be incorporated into the mask itself for simplicity. In this example, the composition no longer uses logical indexing; instead, the entire ROI is a weighted average, where the mask itself is a weighting map.
fg = imread('image.png');
bg = imread('peppers.png');
fg = padarray(fg,[1 1],0,'both');
mask = fgopacity*imadjust(imgaussfilt(im2double(mask),1),[0.5 1]);
roirows = offset(1):offset(1)+size(fg,1)-1;
roicols = offset(2):offset(2)+size(fg,2)-1;
roi = bg(roirows,roicols,:);
roi = uint8(mask.*double(fg) + (1-mask).*double(roi));
out(roirows,roicols,:) = roi;
Attention needs to be paid to image and mask class all throughout these operations.
Nonstandard Tools
The composition process itself can be simplified greatly with basic tools from MIMT. The function replacepixels() simply does multiplicative composition of two images and a supplied mask. fg = imread('image.png');
bg = imread('peppers.png');
padse = imsize(bg,2)-imsize(fg,2)-offset;
fg = addborder(fg,[offset(1) padse(1) offset(2) padse(2)]);
mask = fgopacity*imadjust(imgaussfilt(im2double(mask),1),[0.5 1]);
out = replacepixels(fg,bg,mask);
Alternatively, imblend() can do both blending and compositing of I/IA/RGB/RGBA images. It's trivial to incorporate the mask into the FG to make an RGBA image. The support for RGBA images in MIMT opens the door to simplified workflows. Since imblend() supports scalar opacity as a parameter, the opacity does not even need to be incorporated into the mask. Note that while the output is RGBA, the BG is solid, so the alpha channel can be discarded afterwards. Fair warning, no tools in base MATLAB or Image Processing Toolbox (e.g. imshow(), imwrite()) can handle RGBA images in this form, so you'll have to decouple or discard alpha content when it comes time to write to disk.
fg = imread('image.png');
bg = imread('peppers.png');
padse = imsize(bg,2)-imsize(fg,2)-offset;
fg = addborder(fg,[offset(1) padse(1) offset(2) padse(2)]);
mask = imadjust(imgaussfilt(im2double(mask),1),[0.5 1]);
fg = cat(3,fg,im2uint8(mask));
out = imblend(fg,bg,fgopacity,'normal');
Of course, imblend() is comprehensive, supporting well over a hundred blend modes and several compositing modes, most of which have been parameterized to allow for further flexibility.
out = imblend(fg,bg,fgopacity,'overlay',2);
out = imblend(fg,bg,fgopacity,'equivalence',1);
That's probably enough for now.