It sounds as if FreeImage is leaking memory or memory gets fragmented in the code calling FreeImage. Early in the lifetime of your process, enough contiguous memory is still available and everything works fine. Later on, things start to fail on larger color images. The reason why specifically jpegs are failing is because the memory image of a color image always takes at least 24 bits per pixel (probably 32 bits), even if its size on disk is much smaller. Your tiff images are probably black and white and then require only 1 bit per pixel.
If not all memory for each processed image is released (either in FreeImage.dll or in your own program), that creates a problem in a continuously running process, even if the amount of memory that stays in use is tiny. The problem is that memory will become fragmented, and the OS cannot relocate the fragments in the memory space of a single process. For loading large images, large contiguous blocks of memory are required, and in fragmented memory there may not be large enough free "holes" anymore, even if the overall free space is still enough. A simple workaround is to spawn a separate process for each image to be processed and have the main process only doing task management. This however makes your program slightly more complex and may cause stability problems if it is possible for a worker process to hang.