How to apply base2 logarithm to an RDD of Ints in Spark?

Question

What is the proper way of doing this to apply log2 to my RDD of numbers? Is there a function to help with this?

See this first: http://stackoverflow.com/help/how-to-ask – maasg Nov 25 '16 at 22:02 — maasg, Nov 25 '16 at 22:02

score 1 · Accepted Answer · answered Nov 25 '16 at 22:01

RDD:

import org.apache.commons.math.util.MathUtils

val rdd: RDD[Double] = ???
rdd.map(x => MathUtils.log(2.0, x))

DataFrame:

import org.apache.spark.sql.functions.log2

rdd.toDF("value").select(log2("value"))

1 Answers1