Select values from a dataframe column

Question

I would like to calculate the difference between two values from within the same column. Right now I just want the difference between the last value and the first value, however using last(column) returns a null result. Is there a reason last() would not be returning a value? Is there a way to pass the position of the values I want as variables; ex: the 10th and the 1st, or the 7th and the 6th?

Current code Using Spark 1.4.0 and Scala 2.11.6

myDF = some dataframe with n rows by m columns

def difference(col: Column): Column = { last(col)-first(col) }

def diffCalcs(dataFrame: DataFrame): DataFrame = { import hiveContext.implicits._ dataFrame.agg( difference($"Column1"), difference($"Column2"), difference($"Column3"), difference($"Column4") ) }

When I run diffCalcs(myDF) it returns a null result. If I modify difference to only have first(col), it does return the first value for the four columns. However, if I change it to last(col), it returns null. If I call myDF.show(), I can see that all of columns have Double values on every row, there are no null values in any of the columns.

I pass the column I want to a function that does: `last(col)-first(col)`, but it returns a `null`. The `first(col)` returns the appropriate, but `last(col)` returns `null`. — the3rdNotch, Sep 28 '15 at 16:33
I guess I also want to make clear that none of the values in the column are `null`, they are all doubles. — the3rdNotch, Sep 28 '15 at 17:24
Please, post a [a Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve). — zero323, Sep 28 '15 at 18:40

score 0 · Accepted Answer · answered Oct 09 '15 at 16:03

After updating to Spark 1.5.0, I was able to use the code snippet provided in the question and it worked. That was what ultimately fixed it. Just for completeness, I have included the code that I used after updating the Spark version.

def difference(col:Column): Column = {
  last(col)-first(col)
}

def diffCalcs(dataFrame: DataFrame): DataFrame = {
  import hiveContext.implicits._
  dataFrame.agg(
    difference($"Column1").alias("newColumn1"),
    difference($"Column2").alias("newColumn2"),
    difference($"Column3").alias("newColumn3"),
    difference($"Column4").alias("newColumn4")
  )
}

Select values from a dataframe column

1 Answers1