0

we failed to build streaming cube in first step:save data from kafka, this is the output log:

Counters: 12 
        Job Counters 
                Failed map tasks=4 
                Launched map tasks=4 
                Other local map tasks=3 
                Rack-local map tasks=1 
                Total time spent by all maps in occupied slots (ms)=10836 
                Total time spent by all reduces in occupied slots (ms)=0 
                Total time spent by all map tasks (ms)=10836 
                Total vcore-seconds taken by all map tasks=10836 
                Total megabyte-seconds taken by all map tasks=11096064 
        Map-Reduce Framework 
                CPU time spent (ms)=0 
                Physical memory (bytes) snapshot=0 
                Virtual memory (bytes) snapshot=0

can anyone tell me how to solve it ?

Aaqib
  • 9,942
  • 4
  • 21
  • 30
dujijun
  • 23
  • 1
  • 1
  • 4

1 Answers1

0

I have met the same problem,when I run kylin streaming cube build. When I open the yarn log I saw

Error: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.kylin.source.kafka.hadoop.KafkaInputFormat.createRecordReader(KafkaInputFormat.java:107) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:515) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)`

Following the instructions here, I set KAFKA_HOME and then restarted kylin. There is no such log.

Max von Hippel
  • 2,856
  • 3
  • 29
  • 46
sun sky
  • 1
  • 1