0

I was thinking of writing a simple agent based framework using Hadoop. This is an idea I have been toying around for quite a while now (but have not put a lot of deep thinking into).

The target application I want to use it for is Genetic Programming. The Idea is that I would spawn a lot of agents, have them perform a task, evaluate their genetic fitness and then breed them, and then repeat the whole exercise.

I wanted to know if folks had any immediate thoughts/critique of doing this using Hadoop.

Thanks in advance.

user1172468
  • 5,306
  • 6
  • 35
  • 62

2 Answers2

2

I think you should look into YARN of hadoop 2.0. It enables you to run any arbitrary logic not restricting you to MR paradigm

David Gruzman
  • 7,900
  • 1
  • 28
  • 30
1

It depends on how your agents really fit map/reduce approach. If results produced by agents will be consumed in reduce phase and new jobs will be initiated - you're ok with Hadoop. Buf if your agents depend on results of each other - you probably want to try out something like Storm ( https://github.com/nathanmarz/storm/wiki )

jdevelop
  • 12,176
  • 10
  • 56
  • 112