-1

I'm new to SQL. I have a very large dataset that I query, and am trying to get the rows for many different date ranges, and export these to excel. The Dataset is really large, too large to load into pandas. My question is, is there any way I can save the results of a query in python, so I can run queries on that (much smaller) table rather than having to re-fetch from that dataset? (I do joins and stuff so it takes minutes to load)

Thanks!

Mark Jones
  • 1
  • 1
  • How would you manage such a "huge" data set in Excel? If you have to move data from Vertica to Excel you can use ODBC directly from Excel... Why do you need Python? – mauro Jun 11 '17 at 02:53

1 Answers1

0

Ideally, I would ask the database administrator to grant me the privilege of creating tables in my own little schema.

Then, instead of a heavy SELECT * FROM <whatever>, I would go CREATE TABLE my_stuff /* or a better name */ AS SELECT * FROM <whatever> .

From then on, you have your smaller table and can select from it as long as you need.

And I would like to add: as long as you remain tabular, you can do virtually anything you "normally" do in Excel, in a Vertica Table, using SQL. That would also be considerably faster, as it runs within the database.

If you're disciplined, you drop it again as soon as you don't need it any more ...

Good luck - Marco

marcothesane
  • 6,192
  • 1
  • 11
  • 21