1

I am writing and distributed image processing application using hadoop streaming, python, matlab, and elastic map reduce. I have compiled a binary executable of my matlab code using the matlab compiler. I am wondering how I can incorporate this into my workflow so the binary is part of the processing on Amazon's elastic map reduce?

It looks like I have to use the Hadoop Distributed Cache?

The code is very complicated (and not written by me) so porting it to another language is not possible right now.

THanks

josephmisiti
  • 9,862
  • 11
  • 56
  • 72

2 Answers2

0

The following is not exactly an answer to your Hadoop question, but I couldn't resist not asking why you don't execute your processing jobs on the Grid resources? There are proven solutions for executing compute intensive workflows on the Grid. And as far as I know matlab runtime environment is usually available on these resources. You may also consider using the Grid especially if you are in academia.

Good luck

Shayanlinux
  • 301
  • 1
  • 4
  • the reason i am not using the grid resource is because this port to hadoop is the first step to converting the code to java or python and then implementing the entire system in a different language. – josephmisiti Nov 05 '10 at 14:05
0

Joseph,

I've just asked a similar, but more general version of your question. Hopefully we'll get some answers. ;)

Hadoop Streaming: Mapper 'wrapping' a binary executable

  • Nick
Community
  • 1
  • 1
Nick Crawford
  • 5,086
  • 2
  • 21
  • 20