0

I have a dataset with approximately 100 million rows and almost 10 GB of data stored in the Tableau Cloud. I encounter issues when performing a full refresh, as it takes more than two hours and often results in errors.

I am considering using an incremental refresh; however, I face challenges due to event latency issues with Firebase/BigQuery. Firebase updates my data for the last three days, whereas Tableau's incremental refresh only appends the latest day's data. This leads to missing updates if there are changes in the data from the days before the last one.

One workaround would be to configure the incremental refresh to add data from three days ago, ensuring that the Firebase latency is accounted for. However, this means I won't be seeing the most recent data as quickly as I'd like.

Is there a way to append incremental refresh data from before the last three days, while performing a full refresh only for the last three days' worth of data?

Axis
  • 2,066
  • 2
  • 21
  • 40
  • As I understand the latency problem is that firestore updates every 3 days. You can write a writeback script that instantly updates your firestore data from whereever the data is coming from. Then you will only be worrying about tableau's refresh. – Ziya Mert Karakas Aug 26 '23 at 10:46
  • Hi @ZiyaMertKarakas, thank you for the comment. Even if the Firestore data becomes real-time, we'll still need to handle Tableau's refresh settings. Depending on the frequency, we may still experience a delay in viewing the most recent data. – Axis Aug 26 '23 at 10:52
  • in that case, Let me ask you what is the latency do you require? do you want almost-real time latency? – Ziya Mert Karakas Aug 26 '23 at 10:53
  • If the 3-day delay is built into Firestore, a write-back script will not solve the problem. However, if the delay is due to our own installation, a script that immediately updates Firestore may help to reduce the delay in Tableau. But since we already get most data (>95%) in less than 24 hours, the latency could be an issue with online/offline players data or the network issue. – Axis Aug 26 '23 at 11:03
  • Where is the data dropping into firestore from? – Ziya Mert Karakas Aug 26 '23 at 11:10
  • I am not sure about that one but I think it comes directly from mobile application – Axis Aug 26 '23 at 11:17
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/255064/discussion-between-axis-and-ziya-mert-karakas). – Axis Aug 26 '23 at 11:22

0 Answers0