How to get a "moving query" with firebase

Question

I have data stored with timestamps as IDs. I need to only get new data when the data passes the where query clause. The data also has the same timestamp in a key named day. The code is written in Golang but it only gets the data with a static timestamp from when the function is first called. How do I get the most recent data with a query that moves?

Query


snapshots := org.firestoreClient.Collection(
     fmt.Sprintf("metrics/%v/%v", "id", "data-collectioin"),
).Where("day", ">=", time.Now().UTC().Add(time.Duration(-7) * time.Hour * 24))
.Snapshots(context.Background())

// Listening loop does not terminate
for {
    snapshot, err := snapshots.Next()
    if err == iterator.Done {
        break
    }
    if err != nil {
        log.Println(err)
        continue
    }

    var decodingErr error
    // Doc Iterating loop terminates at the last document
    for {
        store, vErr := snapshot.Documents.Next()
        if vErr != nil {
            break
        }
        // unmarshal data to result struct
        // ....
        // check for old data 
        if result.Day.UTC().Before(time.Now().UTC().Add(time.Duration(-7) * time.Hour * 24)) {
            log.Println("has old data")
            continue
        }
        // save to cache
        addToSlice()
    }
    saveToCache(slice)
}

I expect new data every 24 hours, after which the server running this query will get new data. and save the data to the cache

Maybe my understanding is wrong but 14 days after this query is run, the cache-store will have data from 14 days ago not just data from the last 7 days. I would like to only have data from the last 7 days. The if (old-data) continue skips old data but with a large dataset and given that I only need entries that pass the initial query (absolute), that will be a lot of skipping over data I don't need

Output
day 0:

day 1: 
  has old data
day 2:
  has old data
  has old data
day 3:
  has old data
  has old data
  has old data

score 0 · Accepted Answer · answered Sep 14 '20 at 14:51

0

Firestore doesn't have any dynamic parameters or something like that. If you want to request data for a new timestamp, you'll have to run a new query with the new timestamp.

Most applications thus run a periodic query to clean up expired data.

answered Sep 14 '20 at 14:51

Frank van Puffelen

565,676
79
828
807

score 0 · Answer 2 · answered Sep 15 '20 at 20:32

I found a way to use a function to archive the effect I wanted

query := func () firestore.Query {
    return org.firestoreClient.Collection(
        fmt.Sprintf("metrics/%v/%v", "id", "collection"),
    ).Where("day", ">=", time.Now().UTC().Add(time.Duration(-7) * time.Hour * 24))
}

// Listening loop does not terminate
for {
    snapshot, err := query().Snapshots(context.Background()).Next()
    // ...
    if cacheIsEmpty {
        // initial data
        // init cache
        addDataToCache(snapshot.Documents))
    } else {
        // clear cache
        // re-run query with new timestamp
        addDataToCache(query().Documents(context.Background()))
    }
}

How to get a "moving query" with firebase

2 Answers2

Linked