3

I am brand new to pyspark and want to translate my existing pandas / python code to PySpark.

I want to subset my dataframe so that only rows that contain specific key words I'm looking for in 'original_problem' field is returned.

Below is the Python code I tried in PySpark: 

def pilot_discrep(input_file):

    df = input_file 

    searchfor = ['cat', 'dog', 'frog', 'fleece']

    df = df[df['original_problem'].str.contains('|'.join(searchfor))]

    return df

When I try to run the above, I get the following error:

AnalysisException: u"Can't extract value from original_problem#207: need struct type but got string;"

mayank agrawal
  • 2,495
  • 2
  • 13
  • 32
PineNuts0
  • 4,740
  • 21
  • 67
  • 112

1 Answers1

8

In pyspark, try this:

df = df[df['original_problem'].rlike('|'.join(searchfor))]

Or equivalently:

import pyspark.sql.functions as F
df.where(F.col('original_problem').rlike('|'.join(searchfor)))

Alternatively, you could go for udf:

import pyspark.sql.functions as F

searchfor = ['cat', 'dog', 'frog', 'fleece']
check_udf = F.udf(lambda x: x if x in searchfor else 'Not_present')

df = df.withColumn('check_presence', check_udf(F.col('original_problem')))
df = df.filter(df.check_presence != 'Not_present').drop('check_presence')

But the DataFrame methods are preferred because they will be faster.

pault
  • 41,343
  • 15
  • 107
  • 149
mayank agrawal
  • 2,495
  • 2
  • 13
  • 32
  • 2
    change like to `rlike` – jxc May 18 '18 at 17:11
  • @PineNuts0 look at the edited answer- [`pyspark.sql.Column.rlike()`](http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.Column.rlike) supports regular expression patterns. – pault May 18 '18 at 17:18