Use the memory limit option.
Option usage shown here:
http://www.imagemagick.org/script/command-line-options.php#limit
-limit type value
Set the pixel cache resource limit.
Choose from: area
, disk
, file
, map
, memory
, threads
, or time
.
The value for file
is in number of files. The other limits are in bytes. Define arguments for the memory, map, area, and disk resource limits with SI prefixes (.e.g 100MB).
By default the limits are 768 files, 3GB of image area, 1.5GiB memory, 3GiB memory map, and 18.45EB of disk. These limits are adjusted relative to the available resources on your computer if this information is available. When any limit is reached, ImageMagick fails in some fashion but attempts to take compensating actions, if possible. For example, the following limits memory:
-limit memory 32MiB -limit map 64MiB