You gave it a multiplication. So it gets the result and tries to create a square matrix with as much rows and columns as the result.
what you want is
A=zeros(1187550,391275)
Next time try help $command$
to get an explanation for the function in matlab
or doc $command
for the documentation
edit:
You also need 3.5TB of RAM for that as
1187550*391275*64bit/8(bit per byte)/1024(byte per kB)/1024(kB per MB)/1024(MB per GB) = 3462GB
.
I guess you dont have that much^^
So you should break it down to partial solutions if possible. And if it suits you, you can also use another datatype to reduce the needed memory (single/float would cut it in half).
This problem should be present in any other program as long it allocates the memory for the whole Matrix at once.