Questions tagged [snowflake-pipe]
41 questions
6
votes
1 answer
snowflake pipe status numOutstandingMessagesOnChannel
I have a pipe that seems to be set up fine, but it just isn't working.
I ran
select system$pipe_status('"MY_DB"."MY_SCHEMA".MY_PIPE_NAME');
i'm getting back a growing number of numOutstandingMessagesOnChannel
Can someone please explain what that…

vbp13
- 1,040
- 1
- 10
- 20
6
votes
2 answers
How to Move or Alter a Pipe without missing or duplicating any records
This page on managing pipes suggests a process for altering the copy into statement in a pipe.
Pause the pipe (using ALTER PIPE … SET PIPE_EXECUTION_PAUSED=true).
Query the SYSTEM$PIPE_STATUS function and verify that the pipe execution state is…

David Garrison
- 2,546
- 15
- 25
5
votes
1 answer
How to get Data from a Mysql Database to Snowflake
is there any clever way to get my data from a mysql datatbase into snowflake?
I found two possible ways so far:
Option 1: Put a Snowpipe ontop of the mysql database and the pipeline converts the data automatically.
Option 2: I convert tables…

NicBeC24
- 47
- 1
- 4
4
votes
2 answers
Snowflake Alert Long Running Queries
How to alert long running queries, to multiple users in snowflake ?
Right now the alert is sent only to the account admin role user.
Is there any way to notify the long query alert to "the user running the query OR notify to
multiple users belong…

Sundar
- 95
- 1
- 13
2
votes
1 answer
Clear or truncate information_schema.copy_history table: Snowflake
How to clear the table contents from information_schema.copy_history
history values so that i can use re-use the same values for copy...strong text

Sreeram
- 21
- 3
2
votes
2 answers
How to troubleshoot Snowpipe auto ingest failures?
I'm trying to load all new files from an AWS S3 bucket depending on its paths to two Snowflake tables, but I couldn't yet succeed even with one table.
What I've tried:
Created a stage:
CREATE or replace STAGE DATA_SCIENCE.INFRA.jobs_resource_usage…

user582175
- 954
- 12
- 17
1
vote
1 answer
Access to Snowflake Internal Stage for Non Owner Role
I am putting data into Internal Table Stage and then use Copy command to load into Actual Table.
It is working fine with my ID since I am table owner ( my role).
Now, I am trying to run this process with a different user who has read and write…

PythonDeveloper
- 289
- 1
- 4
- 24
1
vote
1 answer
Snowflake - Fail COPY INTO (Can't parse '0' as date with format 'YYYYMMDD')
My pipe is executing a COPY INTO command every time a parquet file is loaded into a STAGED location in AWS S3, that's working just fine (the execution).
This is my copy query: (summarized)
copy into table_name
from (
TRY_TO_DATE(
…

Emilio Perea
- 31
- 1
1
vote
2 answers
Custom Sink to Snowflake using Snowflake JDBC driver is very slow
I am using Spring Cloud Data Flow to create a custom stream to load data into Snowflake. I have written a custom sink to load data into Snowflake using Snowflake's JDBC driver. The methodology that I used is similar to any database update using the…

user12593294
- 43
- 4
1
vote
0 answers
PySpark Streaming, when writing producing error
I currently have a script that takes data from S3 bucket and produces a data frame. I want to send the data frame to snowflake. Currently I am doing it like this
df_write = df \
.write\
.format(SNOWFLAKE_SOURCE_NAME)\
…

alex
- 51
- 5
1
vote
0 answers
Snowflake ON_ERROR=CONTINUE abort the COPY command for file
Snowkflake documentation for COPY INTO command states (for COPY options)
ON_ERROR = CONTINUE | SKIP_FILE | SKIP_FILE_num | SKIP_FILE_num% | ABORT_STATEMENT
Continue loading the file. The COPY statement returns an error message
for a maximum of…

Noobie
- 461
- 1
- 12
- 34
1
vote
1 answer
How do we stream a data pipeline for data transfer from snowflake to kafka?
I want to get the transformed data from snowflake to my postgres database. To achieve this I want to move the data from snowflake to kafka and then from kafka to postgres. Not able to figure out how to move data from snowflake to kafka.

Vipul Singhal
- 11
- 1
1
vote
2 answers
How to build a data catalog in Glue for Snowflake?
I am new to snowflake. I am wondering how to build a data catalog for snowflake's data?
I cannot find documentation on that. I can find this post. But it seems that it does not use the catalog function.

JOHN
- 1,411
- 3
- 21
- 41
1
vote
1 answer
Is there a way to reduce the time for the snowpipe load time?
I'm reading this User Guide and it mentions that "typically" snowpipe takes 1 minute to load the data. In my experiments I found that it takes a minute always. Where is this 1 minute latency coming from? It feels like there is some batch processing…

TheSPD
- 177
- 1
- 10
1
vote
1 answer
Snowpipe rest api returning always "invalid jwt token"
I'm trying to use Snowpipe rest api as is pointed in Snowflake site:
https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html#data-file-ingestion
I found a python example in here, my code and steps are pretty much the…

Francisco Albert
- 1,577
- 2
- 17
- 34