How does scalding pass user functions to remote MapReduce nodes

Question

When working with Scalding, you have the ability to provide a function. I was wondering how scalding passes these functions to the remote map/reduce tasks? Is this using something in scala or something generic that can be done with anonymous objects?

score 3 · Answer 1 · answered Apr 25 '13 at 21:10

3

It's plain old Java serialization which is done by Cascading.

answered Apr 25 '13 at 21:10

Oscar Boykin

1,974
2
11
16

1

Thanks. Was thinking that but was not sure. – ekaqu Apr 27 '13 at 06:08

score 1 · Answer 2 · answered Mar 07 '13 at 17:52

1

It uses Cascading, a java library upon Apache Hadoop.

answered Mar 07 '13 at 17:52

Alexei Kaigorodov

13,189
1
21
38

How does cascading pass anonymous objects to map/reduce tasks? I know how to do this if the task is given a Class and create it itself, but with an anonymous object it is not as clear. – ekaqu Mar 07 '13 at 18:10

How does scalding pass user functions to remote MapReduce nodes

2 Answers2