HI I am trying to iterate over pyspark data frame without using spark_df.collect() and I am trying foreach and map method is there any other way to iterate?
df.foreach(lambda x: print(x)) and
def func1(x):
firstname=x.firstname
lastName=x.lastName
name=firstName+","+lastName
gender=x.gender.lower()
salary=x.salary*2
return (name,gender,salary)```
```rdd2=df.rdd.map(lambda x: func1(x))```
is there any other way to iterate over data frame
First of all, Spark is not made to do such kind of operations, like printing each records but more on distributed processing. Tune your process to work in a distributed fashion, like in terms of joins - that will unleash the power of spark.
If you want to process each record, UDFs (User defined functions) are a good way to do that. UDFs will be applied to each record once.
we can use this method to iterate over rows
pandasDF = df.toPandas()
for index, row in pandasDF.iterrows():
print(row['itm_mtl_no'], row['itm_src_sys_cd']) ```