Your mask is being divided by the wrong coefficients. You normalize each coefficient by sum(abs(b(:)))
or sum(abs(c(:)))
to ensure that when you filter using convolution masks, the output dynamic range matches the input.
In your case, you need to divide by 6 and not 256. That's why you have a decreased contrast in comparison to what the IPT gives you in MATLAB.
From your previous post, I'll be using this image as it looks like you're using the same one:

Take note that because you didn't specify a threshold for edge
, it figures this out automatically. I actually managed to find the right threshold and it's 0.08995.
Therefore, try this:
%// Read image from StackOverflow
openImage = rgb2gray(imread('https://i.stack.imgur.com/5EJJH.jpg'));
openImage = im2double(openImage); %// Convert to double
%// Corrected masks
b=[-1 -1 -1;0 0 0;1 1 1]/6;
c=[-1 0 1; -1 0 1; -1 0 1]/6;
Gx=abs(conv2(openImage,c,'same'));
Gy=abs(conv2(openImage,b,'same'));
G = sqrt( Gx.^2 + Gy.^2);
out = G > 0.08995; %// Threshold image
figure;
imshow(out);
%// Also show output from edge
figure;
edge(openImage,'Prewitt', [], 'both', 'nothinning');
If we compare them both, we get this:
From your code

From MATLAB's edge
function

They're pretty much the same!