How to perform a client-side full scan in Tarantool?

Question

I need to consistently iterate over entire space in my application. Currently I'm using batches (e.g. classic limit-offset approach), but it can't be done in one transaction and will be inconsistent (e.g. remove from subset of already read tuples will lead to lost tuple, because of changed offset).

Moreover, manual batching is not so user-friendly, I want something like following:

    rows, err := conn.Query(ctx, "SELECT id, title, tags FROM video")
    if err != nil {
        panic(err)
    }
    var videos []Video
    for rows.Next() {
        var v Video
        if err := rows.Scan(&v.ID, &v.Title, &v.Tags); err != nil {
            panic(err)
        }
        videos = append(videos, v)
    }

This is how I can perform SELECT entire table in go and postgresql, which will be internally buffered and still consistent.

Is there a way to achieve this in Tarantool?

score 2 · Accepted Answer · answered Jun 23 '20 at 12:40

May answer maybe is not completely related to your topic, but I try to describe how should be done without SQL.

First of all, Tarantool doesn't support interactive transactions (https://github.com/tarantool/tarantool/issues/2016) and server-side cursors. You can't perform several selects and to be sure that all data are not modified between them.

So, if you satisfied with such restrictions let's continue. Here is a little example. Here I use an official Tarantool go connector (https://github.com/tarantool/go-tarantool)


    spaceName := "tester"
    indexName := "scanner"
    idFn := conn.Schema.Spaces[spaceName].Fields["id"].Id // field number of Id
    bandNameFn := conn.Schema.Spaces[spaceName].Fields["band_name"].Id

    var tuplesPerRequest uint32
    cursor := []interface{}{}
    tuplesPerRequest = 10


    for {
        resp, err := conn.Select(spaceName, indexName, 0, tuplesPerRequest, tarantool.IterGt, cursor)
        if err != nil {
            panic(err)
        }

        if resp.Code != tarantool.OkCode {
            panic(err)
        }
     
        if len(resp.Data) == 0 {
            break // No more data
        }

        tuples := resp.Tuples()
        for _, tuple := range tuples {
            fmt.Printf("\t%v\n", tuple)
        }
        
        lastTuple := tuples[len(tuples) - 1]
        cursor = []interface{}{lastTuple[idFn], lastTuple[bandNameFn]}
    }

I should notice some things. I don't use offset. Never use offsets in Tarantool! I use "cursor". It's a set of fields what you've indexed plus an iterator (in my example Gt). In short "offset" leads to full scan of B-Tree index, while suggested approach specifies a point where scan should be started/continued.

Instead of using Select function, you could use "Call" for your lua stored procedure. E.g. you want to filter tuples directly at database/application server level.

How to perform a client-side full scan in Tarantool?

1 Answers1