How to combine and rgb and a thermal image?
41 views (last 30 days)
I am working with a ZenMuse XT2 camera, and it outputs images in rgb multispectral and in thermal. Both images have be resized into .tiff files with the same size, and I want to create an .rgbt image that has 4 bands on a single image. The images are currently seperate but in the order (visual, thermal). Is there a way to combine each first image to the second image in this way?
edit: RGBA would probably be the better way to combine them
Any help would be appreciated
DGM on 23 Jul 2021
Let's just start with concatenation. We have two test images: an RGB visible image and a single-channel thermal image. How I make them isn't really important, but they can simply be concatenated.
gpict = imread('cameraman.tif'); % raw single-channel source
gradpict = lingrad([256 256 3],[0 1; 1 0],[0 0 1; 1 0 0]*255,'cosine','uint8'); % colored gradient
rgbpict = imblend(gradpict,gpict,1,'moonlight'); % RGB test image
gpict = fliplr(imlnc(gpict,'independent','g',2,'k',3)); % pretend thermal image
rgbtpict = cat(3,rgbpict,gpict); % 4-channel RGBT image
Of course, nothing knows what an RGBT image is. This is just sort of a contrivance of our own design, to fit our own needs. Things like imshow() won't know what to do with it ... of course imshow() doesn't know what an RGBA image is either. Most tools (at least in the MIMT) think a 4-ch image is RGBA. Imwrite() sees a 4-channel image and thinks it's CMYK. All this confusion doesn't mean you can't use these tools to save the data and read it back, but other applications may grossly misinterpret the saved image. The issue of saving a multichannel image is one that I'm not certain I have a good answer for.
Let's start with a simple example of saving the image as RGBA using PNG:
% which is the same as
[readbackrgb,~,readbackalph] = imread('rgbt.png'); % read the image back
readbackrgbt = cat(3,readbackrgb,readbackalph); % concatenate if desired
Note that imwrite/imread really don't support 4-ch RGBA images directly. The fourth channel has to be split off when writing and reconcatenated when reading (if you choose to process the image as a 4-channel image). Of course, other applications are going to treat this image as RGBA and display it accordingly as nonsense:
What happens if you just feed the 4-ch image to imwrite with a TIFF output?
imwrite(rgbtpict,'rgbt.tif') % image is attached; can't embed tiffs
readbackrgbt = imread('rgbt.tif'); % read the whole 4-ch image back
Imwrite treats 4-channel images as CMYK. Consequently, other apps will treat it that way. I'm not sure of an appropriate set of property settings for arbitrary multichannel images.
Of course, TIFF is a container format. You can always just put both images in the same file.
imwrite(rgbtpict(:,:,1:3),'rgbt.tif') % write the RGB image
imwrite(rgbtpict(:,:,4),'rgbt.tif','writemode','append') % append the T image
readbackrgbt = cat(3,imread('rgbt.tif',1),imread('rgbt.tif',2)); % read back both as a 4-ch image
% image is attached as rgbt2.tif
So that's a few ways to save such an image. Like I said, there may be better canonical ways of dealing with arbitrary multichannel images. I'm not terribly familiar with the TIFF tools or dealing with hyperspectral imaging. I don't know how helpful any of this is.
As Image Analyst points out, how you handle/store the image might depend on how you want to visualize it. Juxtapositioning is simple, but may make content colocation less intuitive depending on the amount of detail available in either image. Alpha blending is simple, so let's consider that as a baseline. IA already showed how to do it with regular tools. I'm going to use MIMT tools because I'm lazy.
blendpict1 = imblend(gpict,rgbpict,1,'average',0.5); % 25/75 (FG/BG) blend
blendpict2 = imblend(gpict,rgbpict,1,'average',1); % 50/50 blend with default parameter
blendpict3 = imblend(gpict,rgbpict,1,'average',1.5); % 75/25 blend
Of course, it looks as flat as one should expect from averaging. If all that is needed is an informal means of visualization, the requirements may be more flexible. Besides simple alpha compositing, there are many ways to blend image data. It may be desirable to emphasize different parts of one image or the other instead of combining them so linearly. Just as Photoshop can blend images, so can imblend().
blendpict4 = imblend(gpict,rgbpict,1,'softlight',1);
blendpict5 = imblend(gpict,rgbpict,1,'softerflatlight',1);
blendpict6 = imblend(gpict,rgbpict,1,'meanglow',1); % hybrid of average and max()
blendpict7 = imblend(gpict,rgbpict,1,'meanshadow',1); % hybrid of average and min()
... of course finding the right mode and parameter settings might take some time and discretion, and may depend heavily on the content and contrast of the actual images being used. MIMT does have a multilayer composition GUI (imcompose) and thorough blend mode documentation which may help if blending is considered a useful approach.
Of course, if all that's needed is visualization, there are other ways, such as reducing the visible content to some sort of edge/depth map and then combining that with the thermal image -- sort of like what some IR cameras do for the viewfinder display.
I don't know if any of this is going to be directly helpful, but maybe you can use it to decide how you want to deal with the images and ask further questions.
Some of the tools used in these examples (lingrad, imlnc, imblend) are from MIMT on the FEX:
More Answers (1)
Image Analyst on 22 Jul 2021
Edited: Image Analyst on 22 Jul 2021
To do side by side, do
wideImage = [image1, image2];
To do top and bottom do
tallImage = [image1; image2];
I would think that a monochrome thermal image overlaid on an RGB image would be confusing to interpret and that having the thermal image beside the RGB image would be easier to interpret.
% Get the RGB image
rgbImage = imread('peppers.png');
[rows1, columns1, numberOfColorChannels1] = size(rgbImage);
% Read in gray scale image.
grayImage = imread('cameraman.tif');
[rows2, columns2, numberOfColorChannels2] = size(grayImage);
if rows1 ~= rows2 || columns1 ~= columns2
grayImage = imresize(grayImage, [rows1, columns1]); % Resize to match RGB image.
% Display RGB image
% Display gray scale image as alphadata
h = imshow(grayImage);
h.AlphaData = 0.5;
g = gcf;
g.WindowState = 'maximized'
Misc Overlay links:
Also see attached paper.