1

We are using great_expectations to validate data using Apache Spark. We are unable to validate columns which have the DATE or DATETIME type.

We use the below configuration to check date entries in a table if they are recent or not.

[
                {
                    "expectation_type": "expect_column_min_to_be_between",
                    "kwargs": {
                        "column": "started",
                        "min_value": "2020-12-13"
                         },
                    "parse_strings_as_datetimes": True
                }]

But it throws the error: not supported between instances of 'str' and 'datetime.datetime'

Since str failed, we tried to use

[
            {
                "expectation_type": "expect_column_min_to_be_between",
                "kwargs": {
                    "column": "started",
                    "min_value": datetime.fromisoformat("2020-12-13")
                     },
                "parse_strings_as_datetimes": True
            }]

But here we get the same error: not supported between instances of 'str' and 'datetime.datetime'

We also tried it after removing the parse_strings_as_datetimes argument but it still throws the same error.

We have also tried asking the great_expectations team in slack, but they didn't respond. Does anyone have a clue what is wrong?

Akhil Nambiar
  • 315
  • 3
  • 18
  • Why aren't you trying to parse it after converting to a tabular structure (within a `dataframe`)? With that being said, you can leverage the inbuilt `spark` `date` functions and it would be a much easier option – Dipanjan Mallick May 03 '22 at 17:17
  • The problem is date is always considered as a string by great_expectations. So, when great_expectations module compares date in the dataframe to the `min_value`, it always throws: '>=' not supported between instances of 'datetime.datetime' and 'str' – Akhil Nambiar May 04 '22 at 06:07
  • I've configured with 'datetime' value in the min_value and without 'parse_strings_as_datetimes' param and it works – omriman12 May 22 '23 at 07:21

1 Answers1

2

This appears to be a bug. There is an existing Github Issue open for this, and the internal team will be working on it. I also tried searching the Great Expectations Slack and couldn't find a question like this, so for the future, you may want to confirm that you have actually posted the question in the Support channel of the Slack.

t. gl.
  • 31
  • 2
  • Thanks for linking the github issue to the answer. This problem is exactly what I faced. I have also commented on the github issue. I will just go ahead and implement a custom expectation for the time being. – Akhil Nambiar May 04 '22 at 06:16