4

My goal is to go through my cassandra database, each entry in it, and check a certain field to see if I need to perform an action. I am using the erlang cql driver found here: https://github.com/matehat/cqerl

When I run select * from keyspace.table limit 10; in a table of 2000, I get the first 10 results... but I then need to get the next 1990. How do i get the next 10 using CQL? I see that there is support for paging but no documentation for how I can do it in cql.

I have researched this question extensively and it seems like the only answers on the topic are incomplete and many commenters on those questions did not receive the answer they were looking for: Iterating through Cassandra wide row with CQL3

Thank you for your help in advance!

Code Wiget
  • 1,540
  • 1
  • 23
  • 35
  • 1
    remove `limit 10` and you will get all records. If you want skip some count -use `skip` instead `limit`. –  Oct 04 '17 at 21:52
  • I don't want to load all of them is the point - if i have 10 million records all loaded into my ram at one point i am going to have problems. I want 10, 20, 100, X, amount at a time. Thus, I would be "Iterating" through the table. – Code Wiget Oct 04 '17 at 22:10
  • 1
    oh, I got it. Set `page_size` in `#cql_query` and use code from readme(section Pagination). –  Oct 04 '17 at 22:24
  • @Atomic_alarm This worked perfectly! Thank you. I am going to post an answer about my exact method, I will accept yours when you post one. – Code Wiget Oct 04 '17 at 23:00
  • 2
    no, it would be better if you write own. –  Oct 05 '17 at 05:10

1 Answers1

2

Thanks to @Atomic_alarm, I found the answer.

The docs here : https://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0 specify how to do automatic paging with CQL.

I did not use the cql command line like the docs say, instead I used the erlang driver code. First I inserted 1000 rows into a table, and then I ran the function below. It iterated through all 1000 results, 2 pages at a time:

test_page() ->
       {ok, Client} = cqerl:get_client({}),
       {ok, Res} = cqerl:run_query(Client, #cql_query{statement = "SELECT * FROM dks.devices;",
                       page_size = 2}),
       get_more(Res, 0).
   get_more(Res, Num) ->
       case cqerl:has_more_pages(Res) of
           true ->
               {ok, Res2} = cqerl:fetch_more(Res),
               get_more(Res2, Num+1);
           false -> 
               Num
       end.
Code Wiget
  • 1,540
  • 1
  • 23
  • 35