0

I newbie to Cloudera Hadoop,i want to learn cloudera hadoop administration myself,i am having 12 GB ram installed on windows laptop, for virtual machine around 9 GB available to use, do anyone know how to install and use Cloudera manager in virtual environment to practice cloudera administration.is there any way to work on cloudera manager with only 8 gb ram available?

user2631600
  • 759
  • 1
  • 11
  • 18

4 Answers4

1

So basically first u need to install a Virtual machine with OS i generally prefer CentOS but check for the supported OS by cloudera before installing it and for creating VM you can follow links from internet here is one

http://teaching.idallen.com/cst8207/14f/notes/000_centos_virtualbox_install.html

Then once VM is up and you have internet connectivity on it configure the preriqusites for cloudera mention over here

http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cm_ig_cm_requirements.html

You will have two option manual or automated by cloudera manager proceed with your preferred choice.

http://www.cloudera.com/documentation/enterprise/5-4-x/topics/installation_installation.html

I hope this will help you and others.

Vikas Hardia
  • 2,635
  • 5
  • 34
  • 53
0

You could download cloudera quickstart. Its a vm with many components of cloudera, and requirements is 4gb of ram.

Link:

http://www.cloudera.com/documentation/enterprise/5-3-x/topics/cloudera_quickstart_vm.html

DanielVL
  • 249
  • 1
  • 5
  • 1
    A good alternative is to use a few amazon RHEL/Centos machines instead of a local vm. the quickstart VM is a bit heavy but you will get by with your specs. – Havnar Feb 16 '16 at 11:22
  • If you want pay to amazon.. its a good alternative.. AWS Free tier contains 1GB of ram only. Insufficient for big data technologies.. https://aws.amazon.com/free/?nc1=h_ls – DanielVL Feb 16 '16 at 11:24
  • True that it's not free. However if you want to learn how to install, optimize a real cluster, this is better than the learning VM. (from a unix engineer/cloudera admin point of view) for a dev the VM is a quick way of getting used to the enviornment. But it's still just virtual, adding and removing nodes, decomissioning, kerberos, AD integration, etc etc ... is best learned by doing so on an actual cluster. Alternatively you could find some scrap computers and try to install a cloudera cluster on them. – Havnar Feb 16 '16 at 11:58
  • Hi Hanvar, thanks for your quick reply, i don't know the charges in AWS, it's risk to play around with AWS without knowing about it i think,i planned to practice from scratch from my personal laptop is there any other ways to practice complete cloudera manager cluster setup in virtual ?while surfing i got something like cloudera Vagrant -hadoop cluster ,is it possible to setup cloudera manager using Vagrant?i have no idea about vagrant, if possible do you know how it can be done? – user2631600 Feb 16 '16 at 15:25
0

You can use Vagrant which is a command line utility for managing the lifecycle of virtual machines.

In particular you can follow these instructions: http://blog.cloudera.com/blog/2014/06/how-to-install-a-virtual-apache-hadoop-cluster-with-vagrant-and-cloudera-manager/

Since you don't have a lot of RAM available if you choose to do so, you have to modify the project Vagrantfile to fit your RAM requirements and you may not be able to install all the cloudera services on you nodes.

The default project will spin a cluster of 4 virtual machines; one master node with 4GB of RAM and 3 slaves with 2GB of RAM each.

You can delete one slave from the Vagrantfile, e.g. remove slave3 by deleting the following lines: 

config.vm.define :slave3 do |slave3|

    slave3.vm.box = "precise64"

    slave3.vm.provider :virtualbox do |v|
       v.name = "vm-cluster-node4"
       v.customize ["modifyvm", :id, "--memory", "2048"]
    end

    slave3.vm.network :private_network, ip: "10.211.55.103"
    slave3.vm.hostname = "vm-cluster-node4"
    slave3.vm.provision :shell, :inline => $hosts_script
    slave3.vm.provision :hostmanager

end

So you will have a cluster of 3 nodes; one master with 4GB of RAM ans 2 slaves with 2GB of RAM.

Cecilia
  • 487
  • 3
  • 6
  • 14
0

You can install

  • Install CentOS 6.x or 7.2 from the Centos DVD and choosing "Minimal Server" or "Minimal Workstation" depending if you want a graphical interface
  • Install Cloudera Manager from scratch using the Cloudera Documentation, selecting Path B with external PostgreSQL. It's more representative of the administration tasks than Path A. Path C is just a nightmare to begin with CM.

With 9GB RAM, your cluster can run a little slowly if you had an eclipse instance to develop some proof of concept, but it's acceptable. When your cluster is up, you can stop Cloudera Manager and use Hadoop services as well.

Cheloute
  • 783
  • 2
  • 11
  • 27