1

I am trying to efficiently store and retrieve an array of floats in elasticsearch 6.7. Numeroc doc values are sorted, which means I can't use them directly.

At first I was using the source value of the field, but the performance on a large query is not great.

I tried to encode the float array as binary and decode it inside my script. Unfortunately I'm stuck at converting a byte[4] array to a float in painless.

In Java this would look like this

Float.intBitsToFloat((vector_bytes[3] << 24) | ((vector_bytes[2] & 0xff) << 16) |  ((vector_bytes[1] & 0xff) << 8) |  (vector_bytes[0] & 0xff));

But discarding the sign with & 0xff throws a "Illegal tree structure." in painless.

Any idea on how to do this?

Minimal example:

Setting up the index

# Minimal example binary array
# Create the index
PUT binary_array 
{
  "mappings" : {
      "_doc" : {
          "properties" : {
              "vector_bin": { "type" : "binary", "doc_values": true },
              "vector": { "type" : "float" }
          }
      }
  }
}
# Put two documents
PUT binary_array/_doc/1
{
  "vector": [1.0, 1.1, 1.2],
  "vector_bin": "AACAP83MjD+amZk/"
}
PUT binary_array/_doc/2
{
  "vector": [3.0, 2.1, 1.2],
  "vector_bin": "AABAQGZmBkCamZk/"
}

Sample search to convert the binary array back to the array

GET binary_array/_search
{
  "script_fields": {
    "vector_parsed": {
      "script": {
        "source": """
        def vector_bytes = doc["vector_bin"].value.bytes;
        def vector = new float[vector_bytes.length/4];
        for (int i = 0; i < vector.length; ++i) {
          def n = i*4;
          // This would be the Java way, discarding the sign of bytes 0-2, but is raises a "Illegal tree structure." in painless
          //def intBits = (vector_bytes[n+3] << 24) | ((vector_bytes[n+2] & 0xff) << 16) |  ((vector_bytes[n+1] & 0xff) << 8) |  (vector_bytes[n] & 0xff);
          // This runs but gives incorrect results
          def intBits = (vector_bytes[n+3] << 24) | ((vector_bytes[n+2] ) << 16) |  ((vector_bytes[n+1] ) << 8) |  (vector_bytes[n] );
          vector[i] = Float.intBitsToFloat( intBits );
        }
        return vector;
        """
      }
    },
    "vector_src": {
      "script": """params._source["vector"]"""
    }
  }
}
Victor P.
  • 630
  • 1
  • 6
  • 14

1 Answers1

0

After some more investigation I realized that the bitwise and does work in painless, but the 0xff doesn't.

This solved my issue:

Float.intBitsToFloat( (vector_bytes[n+3] << 24) | ((vector_bytes[n+2] & 255) << 16) |  ((vector_bytes[n+1] & 255) << 8) |  (vector_bytes[n] & 255) )
Victor P.
  • 630
  • 1
  • 6
  • 14