We are working to build a big cluster of 100 nodes with 300 TB storage. Then we have to serve it to different users (clients) with restricted resources limit i.e., we do not want to expose complete cluster to each user. Is it possible ? If it is not possible then what are other ways to do it. Are there any builtin solutions available ? It is just like cluster partitioning on demand.
Asked
Active
Viewed 72 times
0
-
You only have 3TB average of storage on each node? You realize each data node can use more than one hard drive, right? And 4TB+ disks are preferred for this type of storage? I work on a cluster of 100 nodes as well, but it has 5 PB of storage – OneCricketeer Dec 28 '18 at 03:19
-
Are you referring to dividing HDFS data or computing resources? – tk421 Dec 31 '18 at 23:58
-
Computing resources – Hafiz Muhammad Shafiq Jan 01 '19 at 03:37
1 Answers
0
On hadoop 2 there is a concept of HDFS Federation that can partition the file system namespace over multiple separated namenodes each of which manages a portion of the file system namespace.
baba
- 265
- 3
- 9