Native Impala UDF (Cpp) randomly gives result as NULL for same inputs in the same table for multiple invocations in same query

Question

I have a Native Impala UDF (Cpp) with two functions Both functions are complimentary to each other.

String myUDF(BigInt)
BigInt myUDFReverso(String)

myUDF("myInput") gives some output which when myUDFReverso(myUDF("myInput")) should give back myInput

When I run a impala query on a parquet table like this,

select column1,myUDF(column1),length(myUDF(column1)),myUDFreverso(myUDF(column1)) from my_parquet_table order by column1 LIMIT 10;

The output is NULL at random.

The output is say at 1st run as ,

+------------+----------------------+------------------------+-------------------------------------+
| column1    | myDB.myUDF(column1)  | length(myUDF(column1)) | myDB.myUDFReverso(myUDF(column1))   |
+------------+----------------------+------------------------+-------------------------------------+
| 27011991   | 1.0.128.9            | 9                      | 27011991                            |
| 27011991   | 1.0.128.9            | 9                      | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | 14022013                            |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
+------------+----------------------+------------------------+-------------------------------------+

and suppose on the 2nd run,

+------------+----------------------+------------------------+-------------------------------------+
| column1    | myDB.myUDF(column1)  | length(myUDF(column1)) | myDB.myUDFReverso(myUDF(column1))   |
+------------+----------------------+------------------------+-------------------------------------+
| 27011991   | 1.0.128.9            | 9                      | 27011991                            |
| 27011991   | 1.0.128.9            | 9                      | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                | 
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | 14022013                            |
| 14022013   | 1.0.131.239          | 11                     | 14022013                            |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
| 14022013   | 1.0.131.239          | 11                     | 14022013                            |
| 14022013   | 1.0.131.239          | 11                     | NULL                                |
+------------+----------------------+------------------------+-------------------------------------+

And sometimes it gives the correct value for all rows too.

I have tested this on Impala v1.2.4 as well as v2.1 What is the cause of this? Some memory issue?

Edit 1:

BigIntVal myUDF(FunctionContext* context, const StringVal& myInput)
{
  if (myInput.is_null) return BigIntVal::null();

  unsigned int temp_op= 0;
  unsigned long result= 0;
  uint8_t *p;
  char c= '.';

  p=myInput.ptr;

  while (*p != '\0')
  {
    c= *p++;
    int digit= c*2;

    if (digit >= 22 && digit <= 31)
    {
      if ((temp_op= temp_op * 10 - digit) > 493)
      {
        return BigIntVal::null();
      }
    }
    else if (c == '.')
    {
      result= (result << 8) + (unsigned long) temp_op;
      temp_op= 0;
    }
    else
    {
      return BigIntVal::null();
    }
  }

  return BigIntVal((result << 8) + (unsigned long) temp_op);
}

In .h file the macro lowerbytify is defined as 

#define lowerbytify(T,A)        { *(T)= (char)((A));\
                                  *((T)+1)= (char)(((A) >> 8));\
                                  *((T)+2)= (char)(((A) >> 16));\
                                  *((T)+3)= (char)(((A) >> 24)); }

StringVal myUDFReverso(FunctionContext* context, const BigIntVal& origMyInput)
{
  if (origMyInput.is_null)
   return StringVal::null(); 

  int64_t myInput=origMyInput.val;
  char myInputArr[16];
  unsigned int l=0;        

  unsigned char temp[8];
  lowerbytify(temp, myInput);

  char calc[4];
  calc[3]= '.';

  for (unsigned char *p= temp + 4; p-- > temp;)
  {
    unsigned int c= *p;
    unsigned int n1, n2;
    n1= c / 100;
    c-= n1 * 100;
    n2= c / 10;
    c-= n2 * 10;
    calc[0]= (char) n1 + '0';
    calc[1]= (char) n2 + '0';
    calc[2]= (char) c + '0';
    unsigned int length= (n1 ? 4 : (n2 ? 3 : 2));
    unsigned int point= (p <= temp) ? 1 : 0;

    char * begin = &calc[4-length];

    for(int step = length - point;step>0;step--,l++,begin++)
    {
        myInputArr[l]=*begin;
    }
   }

   myInputArr[l]='\0';

   StringVal result(context,l);
   memcpy(result.ptr, myInputArr,l);

    return result;
}

I suspect it's an issue with the memory management. Can you share your UDF code? — Matt, May 08 '15 at 16:23

score 1 · Answer 1 · answered May 08 '15 at 16:46

I don't think you can assume the string is null-terminated. You should use StringVal::len to iterate over the chars rather than while (*p != '\0'). Also, I'd recommend writing some unit tests using the UDF test framework in the impala-udf-samples github, see this example.

Native Impala UDF (Cpp) randomly gives result as NULL for same inputs in the same table for multiple invocations in same query

1 Answers1

Linked