Can not create a max column, with list comprehension. sum column works

Question

I try to create a max column with this code. The sum column works

sum:

for col in list_names:
    for month in [3,6,9,12]:
        companies = companies.withColumn(col + 'sum_'+ str(month) + '_months', sum(companies[col + ult_pats2[month_ix - ix]] for ix in range(month)) )

max:

for col in list_names:
    for month in [3,6,9,12]:
        companies = companies.withColumn(col + 'max_'+ str(month) + '_months', max(companies[col + ult_pats2[month_ix - ix]] for ix in range(month)) )

the error message is:

"ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions"

could you please share the expected input and output you want — Ankit Kumar Namdeo, May 29 '19 at 11:23
[Did you do an `import *`?](https://stackoverflow.com/questions/49481363/pyspark-nonetype-object-has-no-attribute-jvm-error/55711135#55711135) — pault, May 29 '19 at 14:20

score 0 · Answer 1 · answered May 29 '19 at 17:45

0

This looks to me like overwriting the max function by some other package. Try:

import pyspark.sql.functions as f

And then use the reference f.max(...)

answered May 29 '19 at 17:45

michalrudko

1,432
2
16
30

This option -f.max()- gets max value of one column, but it doesn't work with multiple columns. – ecan Aug 21 '19 at 07:06

ecan · Accepted Answer · 2019-08-21T07:17:49.170

0

Finally it worked with this code, using sf.greatest:

import pyspark.sql.functions as sf

for col in list_names:  
    for month  in [3,6,9,12]:
            companies = companies.withColumn('max_'+ col + str(month) + '_months',
                                             sf.greatest( *[sf.col(col + ult_pats2[month_ix - ix]) for ix in range(month)] ) )

edited Aug 21 '19 at 07:17

answered Aug 21 '19 at 07:12

ecan

1
2

Can not create a max column, with list comprehension. sum column works

2 Answers2