4

I'm trying to upload data from a CSV to my app using the devserver:

appcfg.py upload_data --config_file="DataLoader.py" --filename="data.csv" --kind=Foo --url=http://localhost:8083/remote_api "path/to/app"

The result:

Application: appname; version: 1.
Uploading data records.
[INFO    ] Logging to bulkloader-log-20100626.181045
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20100626.181045.sql3
Please enter login credentials for localhost
Email: email@domain.com
Password for email@domain.com:
[INFO    ] Connecting to localhost:8083/remote_api
[INFO    ] Starting import; maximum 10 entities per post
Google\google_appengine\google\appengine\api\datastore_types.py:673: DeprecationWarning: object.__init__() takes no parameters
  super(Link, self).__init__(self, link)
............[INFO    ] Unexpected thread death: WorkerThread-7
[INFO    ] An error occurred. Shutting down...
........[ERROR   ] Error in WorkerThread-1: <urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>
[ERROR   ] Error in WorkerThread-7: <urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>

[INFO    ] 1230 entites total, 0 previously transferred
[INFO    ] 200 entities (218274 bytes) transferred in 25.8 seconds
[INFO    ] Some entities not successfully transferred

The count of entities transferred ranges between 200-850 as I try subsequent times.

What is going on here? Normally this works fine. Navigating to http://localhost:8083/ works, and the app runs just fine. (Except the lack of data.)

Nick Heiner
  • 119,074
  • 188
  • 476
  • 699
  • 2
    Have you tried passing the `--num_threads` option with a small value and see if that helps any? If the machine is refusing connections, perhaps there are too many threads all trying to establish concurrent TCP connections at once (e.g., exceeding the maximum [`backlog`](http://www.opengroup.org/onlinepubs/009695399/functions/listen.html) of the listening server). Worth a try anyway. – David Underhill Jun 27 '10 at 01:37
  • are you perchance using Python 2.6 (rather than the 2.5 which should always be used to run App Engine's local SDK)? That DeprecationWarning looks suspicious... – Alex Martelli Jun 27 '10 at 03:01
  • @David is correct: The dev_appserver is single-threaded, and you're attempting to hammer on it with multiple threads, which eventually causes this. – Nick Johnson Jun 27 '10 at 08:18
  • @David that did solve the problem. I am indeed using Python 2.6 to run the devserver. Is it really that bad? – Nick Heiner Jun 28 '10 at 00:52
  • It will work better with Python 2.5. It will help you catch "bugs" where you accidentally use Python 2.6 syntax (which would fail on the production server). The dev server also runs task queue tasks automatically, but this only works when it is run with Python 2.5. – David Underhill Jun 28 '10 at 01:38

1 Answers1

10

Decrease the number of threads to 4 by adding the command line option --num_threads=4

If it still doesn't work decrease further the number of threads.

mtgred
  • 1,449
  • 14
  • 17