Questions tagged [sqlake]
15 questions
1
vote
1 answer
Upsolver snowflake output creating NULL records in snowflake child table
We have a nested json in our input stream which we are writing into snowflake normalized tables using Upsolver snowflake outputs. The parent table is fine but seeing NULL records in the child table. Why is this happening and how can we solve this…
Ajay C
- 66
- 5
0
votes
1 answer
How do you mask parts of an IP address in a data transformation
I have a data source that includes IP addresses. I want to preserve the structure of the IP addresses in my transformation (do not want to use MD5 to hash them), but I would like to mask certain parts of the IP. For example if my source data…
jhall-upsolver
- 11
- 3
0
votes
1 answer
Using MD5 and missing some records in my output in Upsolver SQLake
I'm losing some data in my output and need help to identify the issue. We create a hashkey using the below 3 columns, rest is pretty much straight forward select from data source Upsert on hashkey.
SET hashkey = MD5(advertiser_id || marketplace_id…
Ajay C
- 66
- 5
0
votes
1 answer
Should I create SYNC jobs only in SQLake?
Should we always be just creating sync jobs as our general rule of thumb in Upsolver SQLake?
Ajay C
- 66
- 5
0
votes
1 answer
Aggregating data in Upsolver and using Athena output to Upsert in Athena
I'm getting Kafka stream which I need to aggregate and load into Athena. As each event comes, the aggregates should update to reflect the new event. I want to re-use this aggregated data for multiple outputs so I used an Upsolver intermediate output…
Ajay C
- 66
- 5
0
votes
1 answer
How can I create an array of key,value pairs within a transformation?
I have some source JSON files that contain {key:value} pairs, for example:
{firstName: "jason", lastName: "smith"}
I would like to take this JSON and create an array of key-value pairs as follows:
[{key: "firstName", value: "jason"},{key:…
jhall-upsolver
- 11
- 3
0
votes
1 answer
In SQLake, if I stop my ingestion copy from job or my intermediate transformation job, what happens to my pipeline
Regular (non sync) Jobs are based on commit time. So if I stop any job, what happens to the dependent jobs. In current version, the entire pipeline would stop. Would Sqlake also behave same way or this would happen only for sync jobs?
Ajay C
- 66
- 5
0
votes
1 answer
Why is our Upsolver Kafka data source trying to connect to broker/node host not defined in connection
We have Kafka data source which has 7 brokers (nodes) lets say node1, node2 to node7. When Upsolver Kafka Data source is created, the connection only specified node1 to node5. node6 and node7 were not added.
When Upsolver is trying to ingest data,…
Ajay C
- 66
- 5
0
votes
1 answer
Can I change the data source for my output jobs in Upsolver
Upsolver output is delayed/stuck - We changed the data source for some outputs to a new one. The odd thing is that the "delay" column in the outputs page seems to point to the old data sources and delay is not changing. Why is this happening,…
Ajay C
- 66
- 5
0
votes
0 answers
How to determine what entities in my buckets are more frequently used by Upsolver?
We want to understand the usage of data inside the buckets so that we can determine the retention time.
We are currently using manual and visual ways from tool to identify the usage
0
votes
1 answer
Why is my Upsolver Kafka data source is stuck and/or not pulling any data
Kafta topic has messages but Upsolver data source is stuck or not pulling any new messages. We have about 15 such data sources and some are working fine but some seem stuck. What is happening?
Ajay C
- 66
- 5
0
votes
1 answer
How do I modify Upsert key for a snowflake output in Upsolver
Is it possible to add an Upsert key to an existing output or change the Upsert key for an existing output without creating a new output?
This is all very hypothetical right now! Since this is new data and a lot of the schemas have been changing…
Ajay C
- 66
- 5
0
votes
1 answer
What lifecycle role can we apply on Upsolver buckets?
We are currently trying to determining approach reducing our S3 storage cost.
Can we get some details round how it can be handled in Upsolver?
We have tried currently in process of identifying data freshness, usability and retention time
0
votes
1 answer
Can I load historical data in Upsolver/SQLake at a later time after current data is already ingested
Ideally historical data is loaded first and then current data but we have current data already loaded into snowflake from Kafka as Upsert outputs. We have to later ingest historical data as well and that will be loaded from diff source, lets says…
Ajay C
- 66
- 5
-1
votes
1 answer