I have multiple tables in a Cassandra keyspace. I want to use Great Expectations to validate my data. I've been trying to use Spark to load data from Cassandra and I was able to create RuntimeBatchRequest using Spark dataframes. However I need to create checkpoints and validate new data as a routine. Can anyone tell how to configure Great Expectations for this purpose? Also I would appreciate any suggestions regarding data validation on Cassandra.
Asked
Active
Viewed 106 times
1
-
do you want to do validation for data that is going to be stored in Cassandra? Or already stored in Cassandra, and you need to validate only new data? Is it possible to make validation before writing to Cassandra? – Alex Ott Nov 14 '21 at 11:57
-
@AlexOtt My data is already in Cassandra. I'm trying to read data from Cassandra and make validations. – alit8 Nov 14 '21 at 12:15