0

I want to send large multiple objects in my JAVA MPI program, and wondering what is the most efficient way to do this?

Currently, my program involves serializing the objects first into bytes before sending byte [] objects to another processor. However, the program seems to slow down when the byte [] object is quite large (~ length of byte [] object is ~ 20 million).

Therefore I was wondering if it would be faster or less memory hungry if I serialized the objects and wrote it to file first? If this is a potential option, could I simply read the file into char[] before sending it another processor?

jnarag
  • 1
  • 1
  • you should first investigate where the performance is dropping. is it in the communication itself ? or is it in the object serialization ? if your object is that big, i guess it contains some (multi dimensions) arrays you might be able to send more efficiently manually instead of relying on "generic" serialization. – Gilles Gouaillardet Aug 21 '19 at 00:49
  • Indeed, I'm sending multiple Integer and Double Arraylist, and I managed to make the communication efficient by sending "chunks" of the byte [] object. However, the slow down in performance is associated with when deserializing the objects. I thought about sending the arrays individually, but as the program grinds to a halt quite early on, I don't think this option will be a long-term solution. – jnarag Aug 23 '19 at 14:56

1 Answers1

0

Most MPI implementations are quite efficient at sending large messages, and should be able to saturate the network if properly configured. Writing the data to a file and reading it back would only introduce additional overhead.

Sourav
  • 379
  • 7
  • 13