Questions tagged [upsolver]
19 questions
1
vote
1 answer
Upsolver snowflake output creating NULL records in snowflake child table
We have a nested json in our input stream which we are writing into snowflake normalized tables using Upsolver snowflake outputs. The parent table is fine but seeing NULL records in the child table. Why is this happening and how can we solve this…
Ajay C
- 66
- 5
0
votes
1 answer
How do you mask parts of an IP address in a data transformation
I have a data source that includes IP addresses. I want to preserve the structure of the IP addresses in my transformation (do not want to use MD5 to hash them), but I would like to mask certain parts of the IP. For example if my source data…
jhall-upsolver
- 11
- 3
0
votes
1 answer
Using MD5 and missing some records in my output in Upsolver SQLake
I'm losing some data in my output and need help to identify the issue. We create a hashkey using the below 3 columns, rest is pretty much straight forward select from data source Upsert on hashkey.
SET hashkey = MD5(advertiser_id || marketplace_id…
Ajay C
- 66
- 5
0
votes
1 answer
Should I create SYNC jobs only in SQLake?
Should we always be just creating sync jobs as our general rule of thumb in Upsolver SQLake?
Ajay C
- 66
- 5
0
votes
1 answer
Aggregating data in Upsolver and using Athena output to Upsert in Athena
I'm getting Kafka stream which I need to aggregate and load into Athena. As each event comes, the aggregates should update to reflect the new event. I want to re-use this aggregated data for multiple outputs so I used an Upsolver intermediate output…
Ajay C
- 66
- 5
0
votes
1 answer
How can I create an array of key,value pairs within a transformation?
I have some source JSON files that contain {key:value} pairs, for example:
{firstName: "jason", lastName: "smith"}
I would like to take this JSON and create an array of key-value pairs as follows:
[{key: "firstName", value: "jason"},{key:…
jhall-upsolver
- 11
- 3
0
votes
1 answer
How do I exclude certain columns within a transformation?
When using Upsolver SQLake, if my source table has 100's of columns, and I want to include most of them in a transformation, but exclude a few, can I do that without having to explicitly map every column in the transformation SQL?
For example, if my…
jhall-upsolver
- 11
- 3
0
votes
1 answer
Upsolver Hive or Athena output have Upsert Partition Fields property. What does this do?
When we create Hive or Athena output in Upsolver, the properties show a Upsert Partition Fields. What does this property really do and should we set it to Yes or No?
Ajay C
- 66
- 5
0
votes
1 answer
Upgrade message pop in Upsolver
Sometimes we see a upgrade message on the UI screen. What does this mean and what happens on click of Upgrade vs Close? Is there any downtime during upgrade?
Ajay C
- 66
- 5
0
votes
1 answer
How do you perform a "one time" data load into Upsolver from a JDCB data source?
I have some historical data that I want to create an Upsolver pipeline with. This data source is an Oracle table, that I connect to with JDBC. I want to bring in a "one time" snapshot of this table, so that I can reference it with Upsolver…
jhall-upsolver
- 11
- 3
0
votes
1 answer
Why is our Upsolver Kafka data source trying to connect to broker/node host not defined in connection
We have Kafka data source which has 7 brokers (nodes) lets say node1, node2 to node7. When Upsolver Kafka Data source is created, the connection only specified node1 to node5. node6 and node7 were not added.
When Upsolver is trying to ingest data,…
Ajay C
- 66
- 5
0
votes
1 answer
Can I change the data source for my output jobs in Upsolver
Upsolver output is delayed/stuck - We changed the data source for some outputs to a new one. The odd thing is that the "delay" column in the outputs page seems to point to the old data sources and delay is not changing. Why is this happening,…
Ajay C
- 66
- 5
0
votes
0 answers
How to determine what entities in my buckets are more frequently used by Upsolver?
We want to understand the usage of data inside the buckets so that we can determine the retention time.
We are currently using manual and visual ways from tool to identify the usage
0
votes
1 answer
Why is my Upsolver Kafka data source is stuck and/or not pulling any data
Kafta topic has messages but Upsolver data source is stuck or not pulling any new messages. We have about 15 such data sources and some are working fine but some seem stuck. What is happening?
Ajay C
- 66
- 5
0
votes
1 answer
How do I modify Upsert key for a snowflake output in Upsolver
Is it possible to add an Upsert key to an existing output or change the Upsert key for an existing output without creating a new output?
This is all very hypothetical right now! Since this is new data and a lot of the schemas have been changing…
Ajay C
- 66
- 5