3

I finally got imbdpy2sql running on Windows 7 with Python 2.7 minGW and pymsql, and it connects to the database fine and creates all the tables but it keeps getting stuck on Flushind MoviesCache and never moves past it. This is its output:

SAVING ImdbID values for ... DONE
DROPPING current database ... DONE
CREATING New tables ... DONE
SCANNING movies

and it lists 10001 and then 20001 and so on until 90001 and then it says * FLUSHING MoviesCache

but it never moves past it. How long should I let it sit there like this? I've left it for at least an hour with no progress if not two hours. Is there anything I can do to get it working? Or is it working and I just need to let it sit for hours and hours?

Also in case this helps i run it by opening command prompt, cding to C:\Python27\Scripts\bin\ and then using this command : 

imdbpy2sql.py -d C:\USers\Jordan\DEsktop\IMDB -u mysql://root:lookool@localhost/IMDB
Bibhas Debnath
  • 14,559
  • 17
  • 68
  • 96
Jordan
  • 141
  • 1
  • 8

2 Answers2

1

No, it's not normal: the flush must take at most some tens of seconds.

I'll try to reproduce it; in the meanwhile, you can try using the CSV import system: run imdbpy2sql.py again with the same options, adding "-c C:\Temp" (no quotes - or something like that: maybe it requires two backslashes, for the directory separator, on a Windows system).

If you have access to a linux/unix environment, you can also do some experiment with partial sets of data, using the reduce.sh script in the docs/goodies/ directory.

Davide Alberani
  • 1,061
  • 1
  • 18
  • 28
  • It ran for close to a week but did eventually finish, I havent played with the data yet though. – Jordan Jan 24 '12 at 01:57
  • Amazing. :-) It's probably something related to pymsql. If anyone knows how to improve its performance, please drop me a note. Thanks! – Davide Alberani Jan 25 '12 at 15:11
0

[MySQL]

In general, if you get an embarrassingly high numbero of "TOO MANY DATA ... SPLITTING" lines, consider increasing max_allowed_packet (in the configuration of your MySQL server) to at least 8M or 16M. Otherwise, inserting the data will be very slow, and some data may be lost.

It's written in their document:)

Glenn Yu
  • 613
  • 1
  • 7
  • 11