0

I am importing data from mongodb into hdfs . I am currently using a PIG script to LOAD data. I need to fetch data from mongodb every 3 hours . For this i need to pass in the mongo.input.query parameter . however i am getting the following exception

java.io.IOException: org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : gte
at org.apache.pig.impl.PigContext.doParamSubstitution(PigContext.java:408)
at org.apache.pig.Main.runParamPreprocessor(Main.java:783)
at org.apache.pig.Main.run(Main.java:577)
at org.apache.pig.Main.main(Main.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

Following is my PIG script

-- First, register jar dependencies
REGISTER mongo-java-driver-3.4.2.jar
REGISTER mongo-hadoop-core-2.0.2.jar
REGISTER mongo-hadoop-pig-2.0.2.jar

SET debug 'on'
SET mongo.input.query '{"createdDate":{ "$gte": { "$date": 1502740800000}, "$lt" : { "$date": 1502751600000} } }'

-- Fetch Data From Mongodb
A = LOAD 'mongodb://localhost:8080/testdb.TESTCOLLECTION' USING    com.mongodb.hadoop.pig.MongoLoader();
DUMP A ;

The query runs fine without the mongo.input.query . However i need to retrieve data based on the created data , for every 3 hours .

Any suggestions ? Alternataives ?

Appreciate the help.

Sid
  • 115
  • 1
  • 12

1 Answers1

0

Turns out it was a formatting issue .

Escaping the '$' solved my problem :

SET mongo.input.query '{"createdDate":{ "\$gte": { "\$date": 1502740800000}, "\$lt" : { "\$date": 1502751600000} } }'
Sid
  • 115
  • 1
  • 12