What is better, read all pages at once or page by page in python-camelot?

Question

I will run camelot on a simple digital ocean instance (1 vCPUs, 1GB ram) everyday to extract information from a PDF with +-150 pages and store in a database. What would be a best practice for this:

a) read all pages at once camelot.read_pdf('file.pdf', pages='all', flavor='stream')?

b) read page by page?

for page in range(150):
   camelot.read_pdf('file.pdf', pages=f'{page}', flavor='stream')

Thanks

You can read this: https://github.com/camelot-dev/camelot/issues/28 Reading all pages at once, probably you can have memory issues. I suggest to process document chunks of a fixed number of pages (25? 50?). — Stefano Fiorucci - anakin87, Apr 29 '20 at 09:40
@modesitt "best" = as I have a limit amount of memory/cpu to use to not shutdown the instance, what would be the proper way to read those big pdf files: all pages at once or page by page. — Murilo Boareto Delefrate, Apr 29 '20 at 16:50
Thanks for the link @Anakin87! That's exactly what I'm looking for! — Murilo Boareto Delefrate, Apr 29 '20 at 16:51

score 3 · Accepted Answer · answered Apr 28 '20 at 21:23

3

You would be able to read them all at once if you had the memory required but you probably don't. Hence, extracting data page by page would probably would be what you consider the "best" since after the data extraction you replace it with the next page meaning each time you read in a new page you free up memory space from the last.

Hope this helped somewhat. :)

answered Apr 28 '20 at 21:23

Zik

101
3

Yes! That's what I'm looking for! The link @Anakin87 gave to me (https://github.com/camelot-dev/camelot/issues/28) answers my question and is exactly what you said! – Murilo Boareto Delefrate Apr 29 '20 at 16:52

What is better, read all pages at once or page by page in python-camelot?

1 Answers1