Questions tagged [snowflake-pipe]

41 questions
6
votes
1 answer

snowflake pipe status numOutstandingMessagesOnChannel

I have a pipe that seems to be set up fine, but it just isn't working. I ran select system$pipe_status('"MY_DB"."MY_SCHEMA".MY_PIPE_NAME'); i'm getting back a growing number of numOutstandingMessagesOnChannel Can someone please explain what that…
vbp13
  • 1,040
  • 1
  • 10
  • 20
6
votes
2 answers

How to Move or Alter a Pipe without missing or duplicating any records

This page on managing pipes suggests a process for altering the copy into statement in a pipe. Pause the pipe (using ALTER PIPE … SET PIPE_EXECUTION_PAUSED=true). Query the SYSTEM$PIPE_STATUS function and verify that the pipe execution state is…
David Garrison
  • 2,546
  • 15
  • 25
5
votes
1 answer

How to get Data from a Mysql Database to Snowflake

is there any clever way to get my data from a mysql datatbase into snowflake? I found two possible ways so far: Option 1: Put a Snowpipe ontop of the mysql database and the pipeline converts the data automatically. Option 2: I convert tables…
NicBeC24
  • 47
  • 1
  • 4
4
votes
2 answers

Snowflake Alert Long Running Queries

How to alert long running queries, to multiple users in snowflake ? Right now the alert is sent only to the account admin role user. Is there any way to notify the long query alert to "the user running the query OR notify to multiple users belong…
2
votes
1 answer

Clear or truncate information_schema.copy_history table: Snowflake

How to clear the table contents from information_schema.copy_history history values so that i can use re-use the same values for copy...strong text
2
votes
2 answers

How to troubleshoot Snowpipe auto ingest failures?

I'm trying to load all new files from an AWS S3 bucket depending on its paths to two Snowflake tables, but I couldn't yet succeed even with one table. What I've tried: Created a stage: CREATE or replace STAGE DATA_SCIENCE.INFRA.jobs_resource_usage…
user582175
  • 954
  • 12
  • 17
1
vote
1 answer

Access to Snowflake Internal Stage for Non Owner Role

I am putting data into Internal Table Stage and then use Copy command to load into Actual Table. It is working fine with my ID since I am table owner ( my role). Now, I am trying to run this process with a different user who has read and write…
1
vote
1 answer

Snowflake - Fail COPY INTO (Can't parse '0' as date with format 'YYYYMMDD')

My pipe is executing a COPY INTO command every time a parquet file is loaded into a STAGED location in AWS S3, that's working just fine (the execution). This is my copy query: (summarized) copy into table_name from ( TRY_TO_DATE( …
1
vote
2 answers

Custom Sink to Snowflake using Snowflake JDBC driver is very slow

I am using Spring Cloud Data Flow to create a custom stream to load data into Snowflake. I have written a custom sink to load data into Snowflake using Snowflake's JDBC driver. The methodology that I used is similar to any database update using the…
1
vote
0 answers

PySpark Streaming, when writing producing error

I currently have a script that takes data from S3 bucket and produces a data frame. I want to send the data frame to snowflake. Currently I am doing it like this df_write = df \ .write\ .format(SNOWFLAKE_SOURCE_NAME)\ …
1
vote
0 answers

Snowflake ON_ERROR=CONTINUE abort the COPY command for file

Snowkflake documentation for COPY INTO command states (for COPY options) ON_ERROR = CONTINUE | SKIP_FILE | SKIP_FILE_num | SKIP_FILE_num% | ABORT_STATEMENT Continue loading the file. The COPY statement returns an error message for a maximum of…
Noobie
  • 461
  • 1
  • 12
  • 34
1
vote
1 answer

How do we stream a data pipeline for data transfer from snowflake to kafka?

I want to get the transformed data from snowflake to my postgres database. To achieve this I want to move the data from snowflake to kafka and then from kafka to postgres. Not able to figure out how to move data from snowflake to kafka.
1
vote
2 answers

How to build a data catalog in Glue for Snowflake?

I am new to snowflake. I am wondering how to build a data catalog for snowflake's data? I cannot find documentation on that. I can find this post. But it seems that it does not use the catalog function.
1
vote
1 answer

Is there a way to reduce the time for the snowpipe load time?

I'm reading this User Guide and it mentions that "typically" snowpipe takes 1 minute to load the data. In my experiments I found that it takes a minute always. Where is this 1 minute latency coming from? It feels like there is some batch processing…
TheSPD
  • 177
  • 1
  • 10
1
vote
1 answer

Snowpipe rest api returning always "invalid jwt token"

I'm trying to use Snowpipe rest api as is pointed in Snowflake site: https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html#data-file-ingestion I found a python example in here, my code and steps are pretty much the…
Francisco Albert
  • 1,577
  • 2
  • 17
  • 34
1
2 3