Stack Overflow
Stack Overflow

Questions tagged [great-expectations]

Great Expectations is an open source software that helps teams promote analytic integrity by offering a unique approach to data pipeline testing. Pipeline tests are applied to data (instead of code) and at batch time (instead of compile or deploy time). Pipeline tests are like unit tests for datasets: they help you guard against upstream data changes and monitor data quality. In addition to pipeline testing GE also provides data documentation/profiling

131 questions
0
votes
1 answer

How to locate and install Great_Expectations .JAR file in Azure Synapse WorkSpace

I'm trying to locate and upload Great_Expectations .JAR file to Azure Synapse via Azure Synapse Studio, to update Apache Spark I would normally upload a requirements.txt manually via Apache Spark Pool 'Packages' but I'm having problems doing that,…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
0 answers

Is it possible to change the Great Expectations logo shown in Great_Expectations data docs

I received tremendous help visualizing Great_Expectations Data Docs from Apache Spark both with Databricks and Synapse see How to Save Great Expectations results to File From Apache Spark - With Data Docs I was wondering if it was possible to…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
1 answer

Great Expectation profiling on SparkDF takes a long time when there are many columns

I need to profile data coming from snowflake in Databricks. The data is just a sample of 100 rows but containing 3k+ columns, and will eventually have more rows. When I reduce the number of columns, the profiling is done very fast but the more…
Francis Gosselin
  • 1
0
votes
1 answer

ValueError: root_directory must be an absolute path: Error when access directory in ADLS from Synapse Workspace

When trying to access ADLS directory with the following PySpark code in Apache Spark I get the error: ValueError: root_directory must be an absolute path. Got abfss://root@adlspretbiukadlsdev.dfs.core.windows.net/RAW/LANDING/ instead. Traceback…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
1 answer

How to run great expectation on docker

I have tried few steps but getting issue. Below is my project structure GE-ToolProject ---great_expectations --checkpoints --expectations -.ge_store_backend_id -demo-suite.json …
varsha
  • 63
  • 5
0
votes
1 answer

How to generate a new error report in Great Exceptions (data validation tools) or access the last one?

I recently became acquainted with Great Expectations, a great data validation library and was able to quickly create my own validation suite for a sample data source and view its output. After creating a checkpoint and running it, only the overall…
smbanaei
  • 1,123
  • 8
  • 14
0
votes
2 answers

Fail to install great expectation. Error : Cannot uninstall 'ruamel-yaml'

When I was trying to install great_expectation on terminal, 'pip install great_expectations' I got this error: Installing collected packages: ruamel.yaml, jsonpatch, altair, great-expectations Attempting uninstall: ruamel.yaml Found existing…
Adamtky
  • 33
  • 4
0
votes
1 answer

Great Expectations custom expectation not ignoring nulls as requested

Versions of the libraries we're using: snowconn==3.7.1 snowflake-connector-python==2.3.10 snowflake-sqlalchemy==1.2.3 SQLAlchemy==1.3.23 great_expectations==0.13.10 pandas==1.1.5 Note we're grabbing data from Snowflake on our own and then feeding a…
Ryan Schuster
  • 494
  • 4
  • 15
0
votes
2 answers

Great Expectations SQLAlchemy doesn't enclose lowercase column names

I am trying to use SQLAlchemy and Great Expectations for testing data quality of datasets stored in Snowflake DB. The dataset is called candidates and the column of interest is called first_name. However, when I run sql_dataset =…
RadRuss
  • 484
  • 2
  • 6
  • 16
-1
votes
2 answers

Great expectations installation to AWS EMR

I tried to use great expectations for data quality purpose I am running my jobs in AWS EMR cluster and I am trying to launch great expectations job on AWS EMR as well I have bootstrap script for installation dependencies on a cluster. It looks like…
Liu Piu
  • 35
  • 1
  • 1
  • 7
1 2 3
…
8
9