-1

I have Cloudera quickstart CDH 5.15 cluster is very slow when i run a simple hadoop command like "hadoop fs -ls" it takes almost 20 seconds but when i try runnnig local commands like "ls" it is very fast please help me with this.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
charan kumar
  • 1

1 Answers1

2

The quickstart VM requires 6-8 GB of RAM to work reliably.

But the JVM startup process for any hadoop command is going to be much much slower compared to other built-in shell commands that operate similarly. There's no way around that fact.

If you want the Hadoop ls command to be quicker, it would be beneficial to setup an actual distributed cluster with adequate memory for the Namenode process, which is what ls contacts

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Already I have allocated 8 GB Ram for that Virtual Machine still it is slow . – charan kumar Oct 31 '18 at 07:27
  • Okay... VMs are just slow in general. It's not clear where you have heard that *it should be fast* – OneCricketeer Oct 31 '18 at 07:31
  • But VM is fast only hadoop services are slow – charan kumar Oct 31 '18 at 07:40
  • It's not clear what you're running inside the VM. For example, if you're only listing files, you aren't required to have Hbase, Hive, Impala, Spark, etc running – OneCricketeer Oct 31 '18 at 07:47
  • In VM when i open hive or hbase or pig or any other hadoop components it takes long time around a minute even hadoop commands like listing files also takes 20 seconds.But VM is fast in terms of opening file browser or terminal or firefox etc – charan kumar Oct 31 '18 at 07:59
  • Again, every single one of those Hadoop components requires a Java process to warm up, and in a constrained memory environment, you are subject to memory swapping, causing slow downs... There is no true fix other than stop unneeded services, give more memory to the VM, or don't use a VM at all, and run Hadoop natively – OneCricketeer Oct 31 '18 at 08:03
  • So increasing swap memory will work ? or only stopping the unneeded solution is the only true ? – charan kumar Oct 31 '18 at 08:12
  • 2
    you can stop most of the services, but the performance may not improve just by extending the swap, generally swapping is bad in hadoop – shaine Oct 31 '18 at 11:27