3

I have a web application and 1 remote clusters(It can be one or more). These cluster can be on different machines. I want to perform following operations from my web application:

1 HDFS Actions :-

  • Create New Directory
  • Remove files from HDFS(Hadoop Distributed File System)
  • List Files present on HDFS
  • Load File onto the HDFS
  • Unload File

2 Job Related Actions:-

  • Submit Map Reduce Jobs
  • View their status i.e. how much job has completed
  • Time taken by the job to finish

I need a tool that can help me do these tasks from the web application - via an API, via REST calls etc. I'm assuming that the tool will be running on the same machine( as the web application) and can point to a particular, remote cluster.

Though as a last option(as there can be multiple,disparate clusters, it would be difficult to ensure that each of them has the plug-in,library etc. installed), I'm wondering if there would be some Hadoop library,plug-in that rests on the cluster,allows access from remote machines and performs the mentioned tasks.

Aziz Shaikh
  • 16,245
  • 11
  • 62
  • 79
user1872420
  • 31
  • 3

1 Answers1

2

The best framework which allows everything you have listed here is Spring Data - Apache Hadoop. This has Java Scripting API based implementations to do the following

1 HDFS Actions :-

Create New Directory
Remove files from HDFS(Hadoop Distributed File System)
List Files present on HDFS
Load File onto the HDFS
Unload File

As well spring scheduling based implementations to do the following

2 Job Related Actions:-

Submit Map Reduce Jobs
View their status i.e. how much job has comleted
Time taken by the job to finish
shazin
  • 21,379
  • 3
  • 54
  • 71
  • hi.....Thanks for the reply......if my remote clusters are on different machines then will this api gives me any methods so that i can run my map reduce jobs/perform hdfs actions on that cluster directly.... – user1872420 Dec 04 '12 at 06:49
  • Yes, check the webinar available on the link so that you can better understand. – shazin Dec 04 '12 at 07:47