0

I am using Pig on Hadoop and DataFu sample here (http://datafu.incubator.apache.org/docs/datafu/guide/set-operations.html), here is my code and error message, anyone have any thoughts what is wrong? Thanks.

register datafu-1.2.0.jar;
define setDifference datafu.pig.sets.SetDifference();

-- ({(3),(4),(1),(2),(7),(5),(6)},{(1),(3),(5),(12)})
input = load 'input.txt' AS (B1:bag{T:tuple(val:int)},B2:bag{T:tuple(val:int)});

differenced = FOREACH input {
  -- input bags must be sorted
  sorted_b1 = ORDER B1 by val;
  sorted_b2 = ORDER B2 by val;
  GENERATE SetDifference(sorted_b1,sorted_b2);
}

-- produces: ({(2),(4),(6),(7)})
DUMP differenced;

[main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve SetDifference using imports:

thanks in advance, Lin

Lin Ma
  • 9,739
  • 32
  • 105
  • 175
  • 1
    Try the following: check if the path to UDF is valid `datafu.pig.sets.SetDifference();` if the path is valid, try to restart Grunt, do you still encounter it? – Flowryn Sep 05 '15 at 05:51
  • @Flowryn, thanks for the advice, what do you mean path to UDF is correct? More details appreciated. – Lin Ma Sep 05 '15 at 05:57
  • 1
    Sorry, I wasn't reading carefully, the problem is in another part. the package is correct and you are using a library here. I must run the code also, now I am not at the office. (Later edit: by 'path' I was meaning 'package', sorry again for confusion) – Flowryn Sep 05 '15 at 08:14

1 Answers1

4

@LinMa : Looks like you are using wrong case while accessing the UDF.

Alias defined to access the UDF is :

define setDifference datafu.pig.sets.SetDifference();

You have to use the alias name while using/invoking the method.

GENERATE setDifference(sorted_b1,sorted_b2);
Murali Rao
  • 2,287
  • 11
  • 18
  • thanks for the advice. I think I already using name "setDifference"? No? – Lin Ma Sep 05 '15 at 21:23
  • 1
    @LinMa : in generate statement I see setDifference method name in upper case, it should be in lowercase as per your define statement. – Murali Rao Sep 06 '15 at 00:59
  • thanks and it seems you are also using upper case -- "datafu.pig.sets.SetDifference();"? And this is why I am confused. :) – Lin Ma Sep 06 '15 at 01:00
  • I got what you mean. Thanks and good to know it is case sensitive when using alias. :) – Lin Ma Sep 06 '15 at 01:16