When you are using imagesc (short for image scale), you are visualizing a matrix by mapping the lowest value of that matrix to one end of a colormap, and the highest value of the matrix to the other end.
By default, MATLAB uses the jet() colormap which is the normal RGB-colorrange. If some parts of your image turns out to be yellow, it means that the elements of the matrix has been some spesific place between the highest and lowest value.
The example below hopefully illustrates this more clearly and shows how you can segment out the "yellow" regions of a matrix (which doesn't really have any color per se)
colorRes = 256;
%# In a jet colormap with size 256, yellow is at placement 159
yellow = 159;
yellowScale = ((yellow/256));
image = repmat(1:colorRes,40,1);
figure(1);clf;
colormap(jet(colorRes))
subplot(2,1,1)
imagesc(image)
title('Original image')
%# Segment out yellow
colorDist = 1/colorRes*5; %# Make scalar higher to include colors close to yellow
imageSegmented = zeros(size(image));
imageSegmented(abs(image/colorRes-yellowScale)< colorDist) = 1;
subplot(2,1,2)
imagesc(imageSegmented)
title('Yellow segmented out')
