18

Where in the sqlContext.implicits._ does it define the $"string" to represent a dataframe call to the parent dataframe's column? Specifically I was confused on seeing something like the following:

import sqlContext.implicits._
df.where($"type".isin("type1","type2") and $"status".isin("completed","inprogress"))
nobody
  • 7,803
  • 11
  • 56
  • 91

1 Answers1

32

If you see the following link

class SQLContext.implicits$.StringToColumn

Converts $"col name" into an Column.

Here is the link for latest version.

zero323
  • 322,348
  • 103
  • 959
  • 935
user2844511
  • 420
  • 5
  • 9
  • And in consumable English, this means that, for instance in the OP's example, `$"status".isin("completed", "inprogress")` means that the column name `"status"` is converted / treated as an object itself, then within that object transformations or actions can be made. In this case, only rows that have the column `"status"` containing either `"completed"` or `"inprogress"` will be kept. Hope that helps! – Mike Williamson Apr 06 '22 at 18:10