A lot of things could make the difference, and there is not enough information here to know what. For example, are they both 64-bit OS's and 64-bit JVMs? What about the behavior of malloc - that's up to the OS. Just because a process asks for N bytes of memory doesn't mean it immediately gets that much memory - memory allocators can be very clever. Then there is the question of whether it's actually an apples-to-apples measurement in terms of how the OS reports it.
"Memory usage" means a lot of things. Are we talking about the Java heap (if you take heap dumps of both VMs after startup and an identical priming bit of work, are they the same size or different?), or that plus class data, etc.? You also have hotspot in the picture, compiling Java bytecode into native code that will be different between the two OS's (maybe very different sizes if your Solaris box is a Sparc machine)
The most likely thing is 64-bit vs. 32-bit, but it's impossible to say. You might use some native profiling tools on each to see what calls are allocating memory - that would start to clarify things.
Unless it's causing a problem, it's probably not something to worry about - but healthy curiosity is a good thing.