7

I have a class that takes a Spark DataFrame and does some processing to it. Here is the code:

    for column in self.sdf.columns:
        if column not in self.__columns:
            row = [column]
            row += '--' * 9
            column_table.append(row)

I have this code inside constructor of my class:

self.sdf: Optional[SparkDataFrame] = None

Here sdf is set dynamically during execution of my class and then the for loop mentioned above will run. __columns is a dictionary which is supposed to have all the columns of sdf. I found no errors in the code when it ran. But when I type checked my code with mypy, it showed an error on the first line of for loop:

error: Item "None" of "Optional[Any]" has no attribute "columns"

I understand that initially sdf will be None. But should I consider this a serious error? Is there are any workarounds for this?

ahrooran
  • 931
  • 1
  • 10
  • 25
  • 1
    I don't pretend to understand why, but for a custom class attribute I stopped mypy giving that error by changing `self.thing: Optional[OtherClass] = None` to `self.thing: OtherClass = None` – binaryfunt Jan 26 '21 at 10:15
  • 1
    Ah, the reason why my commented solution "worked" was because mypy wasn't actually working properly. For that project I needed `mypy --explicit-package-bases --namespace-packages`. So my solution doesn't actually work – binaryfunt Mar 18 '21 at 11:38

1 Answers1

4

Yes, columns is specific to data frame. you can find more info here .When you set self.sdf is None, you will get the error you posted. you can alternatively try this.

    from pyspark.sql import DataFrame

    if self.sdf is not None and isinstance(self.sdf,DataFrame):
        for column in self.sdf.columns:
            if column not in self.__columns:
                row = [column]
                row += '--' * 9
                column_table.append(row)
kites
  • 1,375
  • 8
  • 15