4

I am working in Google App Engine and we have a Python script that dumps data in Google Cloud SQL. One of the data sets we have to dump is huge. We dump around 150K rows of data once a day daily.

I know Google Cloud SQL does not support LOAD DATA INFILE, which I would have normally used. My question is, whether there is an alternative to LOAD DATA INFILE that I can use to speed up the process of data dumping.

Inserting the data normally, without LOAD DATA INFILE, takes about 5 minutes.

Tanvir Shaikh
  • 651
  • 2
  • 7
  • 10

2 Answers2

3

As stated in this comment of another question, LOAD DATA LOCAL INFILE is supported by App Engine.

The MySQL Manual explains on how to use this statement.

Community
  • 1
  • 1
Tex
  • 33
  • 6
  • The linked answer doesn't mention `LOAD DATA LOCAL INFILE`. I think you wanted to link to the comments section of the corresponding question instead. – honk Oct 15 '14 at 18:28
  • Thanks. Yeah that's what I wanted. Is there a way to link to comments. That will be preferable – Tex Oct 17 '14 at 12:12
  • I think your answer is fine now. However, it is possible to link to comments: You can access the link to a comment by right clicking on the timestamp of that comment. In the context menu then select to copy the link target. – honk Oct 17 '14 at 18:55
1

Things you can do to get better bulk import performance:

  • Create a .sql file and do an import
  • Make sure that the insert statements do more than one row at a time. A good rule of thumb is one megabyte per INSERT.
  • Switch to async replication
  • Do the import from an App Engine app. The app will be colocated with your Cloud SQL instance, greatly reducing the network latency.
Ken Ashcraft
  • 559
  • 3
  • 8