0

My question is similar to this one.

I have unknown amount of records to import. I'm generating GET calls with offset and limit to import the data as follows:

 http://www.myweb.com?WithTotal=true&offset=X&limit=Y

X is set by loop while Y is always the same.

lets assume that Y is 100.

This is what I generate:

Generate page: #0     http://www.myweb.com?WithTotal=true&offset=0&limit=100
Generate page: #1     http://www.myweb.com?WithTotal=true&offset=100&limit=100
Generate page: #2     http://www.myweb.com?WithTotal=true&offset=200&limit=100
 ....

I'm not sure this is right because some records are missing.

I calculate as follows:

value_limit = 100
num_of_records = Input from service
num_of_pages = int(math.ceil(num_of_records / value_limit))
pages = [i for i in range(0, num_of_pages)]
for page in pages:
    ....
    value_offset = page * value_limit
    Generate URL
    ....
Programmer120
  • 2,362
  • 9
  • 30
  • 48
  • 1
    You need to be more specific as to what records are missing. Is there a pattern in which ones are excluded? There doesn't seem to be anything wrong with the math. – Grant David Bachman Aug 29 '18 at 10:21
  • @GrantDavidBachman When trying to import 50 pages with limit 100 - I should have 5000 records. But I have only 4500. (The total number of available records is 21610 I limit the page numbers for test so I should get 5000 not 4500) – Programmer120 Aug 29 '18 at 10:48

0 Answers0