2

My Scala/Java code has a large (several hundred megabytes) integer arrays. I would send these as fast as possible (without copying the contents) from this int[] using java.nio. Also, when I receive any data on my nio socket channels, I would use the data as int[] without copying.

All nio operations require ByteBuffer, while ByteBuffer can only wrap byte [].

I can allocate a ByteBuffer, then put my array using asIntBuffer, but it will copy the random array and I have to store my data twice.

    val size = 1234

    val random = Array.fill[Int](size) { scala.util.Random.nextInt(1000) }

    val buf = ByteBuffer.allocate(4 * size)
    buf
      .asIntBuffer()
      .put(random)

I am using project panama openjdk, therefore any new trick (like using Pointer<>) could help me.

TL;DR: In Java or Scala, how can access my integer array as ByteBuffer and int[].

Tamas Foldi
  • 487
  • 4
  • 8
  • Have you looked at the `wrap` method on IntBuffer and ByteBuffer? – user May 25 '20 at 23:29
  • ByteBuffer can only `wrap` byte arrays but not integers. IntBuffer can wrap integers but nio socket channels only accept ByteBuffer – Tamas Foldi May 26 '20 at 02:07
  • 1
    How do you imagine this? You need convert one array of data to another array of data, you can't do this operation better than O(2n) memory consumption. Maybe the right direction to think is distribute your long array to some batches. – Boris Azanov May 26 '20 at 06:23
  • 2
    You don’t have to send the entire int array in one write call. Allocate a ByteBuffer of a reasonable size (like four megabytes), use asIntBuffer.put to fill it with subsections of your int array, and send those subsections one at a time. It’s likely that your network interface will be the bottleneck, not the speed of the program. – VGR May 26 '20 at 12:12
  • My whole point is to use as few copy ops as possible. My ideal scenario is to use `allocateDirect` in *Buffer, and use it for nio zero-copy while accessing its contents. The use case is: I am working on some MPI like message passing interface, while I need sometimes low latency and sometimes high throughput. Coming from other languages, `sendfile()`/zero copy is the way to go. Isn't this the same in JVM? – Tamas Foldi May 26 '20 at 12:37
  • I understand your desire, but bulk copies are quite fast, and I’m pretty sure they’ll execute faster than your network can process them. – VGR May 26 '20 at 13:18

0 Answers0