Google App Engine and Upload script

Question

I'm currently using Eclipse that came with Android Development bundle and following the GAE tutorial:

https://cloud.google.com/developers/articles/how-to-build-mobile-app-with-app-engine-backend-tutorial

I never had any server or backend experiences and I am hoping to make a Android application using features of the App Engine.

Everything seems to be going smoothly following the tutorial until I got to the part where you need to "Create an upload script". I have no idea how I should proceed the more I read into it the more confused I am.

The tutorial said to "Create a directory and make it your current directory" then create a script file but am I suppose to make the new directory and put it in Eclipse's AppEngine project in the package explorer or somewhere else? I don't really know how does all of this work together.

I just got python 2.7 installed for Eclipse and made sure that the PyDev interpreter is pointing to python.exe but besides that I don't really know where to go from that point on.

EDIT1: I am using Cygwin and run "sh upload_data.shPlace". And when I run it I get the following error:

C:\Python27\python.exe: can't open file '/cygdrive/c/Program Files (x86)/Google/google_appengine/appcfg.py': [Errno 2] No such file or directory

I tried adding "C:\Program Files (x86)\Google\google_appengine\appcfg.py" to the upload.sh file. It looked like the following:

C:\Program Files (x86)\Google\google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e myemail@mail.com

The following error occurs:

upload_data.sh: line 2: syntax error near unexpected token `('
upload_data.sh: line 2: `/cygdrive/c/Program Files (x86)/Google/google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com'

EDIT2: Ok I added '\' to before the '(' and used '\' for spaces as well. The following is code in upload.sh right now.

C:\Program\Files\(x86\)\Google\google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

Then the following error it came up with the following error:

upload_data.sh: line 2: C:ProgramFiles(x86)Googlegoogle_appengineappcfg.py: command not found

I went to the google app engine directory and the appcfg.py is there. Additionally I am not too sure if I am actually using the right command. I used:

sh upload.data.sh (arrow bracket)places.csv(arrow bracket)Place

Does the scripts or anything needs to be placed in certain directory? I've tried putting the appcfg.py file in my scripts directory where I keep the upload_data.sh file just said [Errno 2] no such file or directory. Then I tried put the files, upload.data, places.csv and bulkloader.yaml into Google/google_appengine directory where appcfg.py is stored then it will return a "Permission Denied" error. I also re-installed python27 into "google_appengine" file. I tried just running "appcfg.py" on Cygwin and just return [Err2] No such file or directory.

Then I tried running running on Windows command prompt writing in:

appcfg.py upload_data --config_file
C:\Users\User\Documents\AndroidApps\Scripts\bulkloader.yaml--url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

The following error gets returned:

11:09 PM Uploading data records.
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 126, in
<module>
    run_file(__file__, globals())
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 122, in
run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4934, in <module>
    main(sys.argv)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4925, in main
    result = AppCfgApp(argv).Run()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2648, in Run
    self.action(self)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4605, in __call__
    return method()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4417, in PerformUpload
    run_fn(args)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4298, in RunBulkloader
    sys.exit(bulkloader.Run(arg_dict))
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bu
lkloader.py", line 4406, in Run
    SetupLogging(arg_dict)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bu
lkloader.py", line 4362, in SetupLogging
    file_handler = logging.FileHandler(log_file, 'w')
  File "C:\Program Files (x86)\Google\google_appengine\lib\logging\__init__.py",
 line 889, in __init__
    StreamHandler.__init__(self, self._open())
  File "C:\Program Files (x86)\Google\google_appengine\lib\logging\__init__.py",
 line 908, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: 'C:\\Program Files (x86)\\Google\\google_
appengine\\bulkloader-log-20140314.230944'

It seems like I can't use appcfg.py in Cygwin at all. I've been stuck for almost 2 days now.

Thanks.

Martin Berends · Answer 1 · 2014-03-14T22:35:17.510

1

The linked tutorial does not say so, but the "Create Upload Script" section seems to be limited to uploading data from a .csv file to the development server. My interpretation is that the directory can be anywhere and has not relation to other projects in the tutorial.

When the tutorial says "Create a new directory and make it your current directory" this implies opening a command shell in a text console. The tutorial assumes you are using a Unix type system, not Windows. Imagine the directory will be called mobile-assistant-upload and that you use vi as your text editor, then your shell commands would be:

mkdir mobile-assistant-data
cd mobile-assistant-data
vi upload_data.sh

Note that all the text from appcfg.py to the email address must all be in one long line, and that you should use your own email address.

EDIT: I think line 2 of upload_data.sh should look more like:

/cygdrive/c/Program\ Files\ \(x86\)/Google/google_appengine\appcfg.py --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

and the command line to run it:

sh upload.data.sh places.csv Place

without arrow brackets, they were just documentation. Of course this assumes that places.csv is in the current directory.

edited Mar 14 '14 at 22:35

answered Mar 13 '14 at 15:18

Martin Berends

3,948
2
16
19

I created the script in a directory in "My Documents" and I installed Cygwin in order to run the upload_data.sh file. The problem I am having now is that when I try to run it I get an error – user1832478 Mar 13 '14 at 16:19
1

appcfg.py is an AppEngine SDK script whose directory must be in your search path. And I warned you to put the --config-file and your own email address on the same line as appcfg.py. – Martin Berends Mar 13 '14 at 16:42
thanks I already put the --config-file and my email address on the same line. How do you put App Engine SDK in the search path? I've tried adding "C:\Program Files (x86)\Google\google_appengine" to environment variables and I tried adding it to python path but I have no idea if what I am doing is right because it is still showing that python.exe can't open file error. – user1832478 Mar 14 '14 at 00:23
1

Now you have a Cygwin problem. The [Using Cygwin](http://cygwin.com/cygwin-ug-net/using.html#pathnames-win32) native path notes or the rest of that doc might help. You could also try not altering the search path but rather editing the appcfg.py name in upload_data.sh to have its full path in the command line, if you know what I mean. Possibly it could be "C:\Program Files (x86)\Google\google_appengine\appcfg.py" although I cannot verify that from outside your computer.. – Martin Berends Mar 14 '14 at 04:32
Hi I updated my question. I not sure if thats what you mean in your comment. – user1832478 Mar 14 '14 at 09:23
1

You're getting closer! Now the shell does not understand that the ( and ) characters are merely literals in a directory name. You might be able to fix that by escaping them, for example by putting \ characters in front of them. The same applies to the spaces in that directory name - next try prefixing the spaces with \ characters as well. Whoever decided to allow spaces and other metacharacters in file and directory names (and then built that into operating systems) should be severely punished for wasting huge amounts of peoples' time. – Martin Berends Mar 14 '14 at 10:29
Thanks for your help so far. I've made further changes to my scripts and edited my question. Now another error strikes: command not found. – user1832478 Mar 14 '14 at 22:10
The ' . ' in "upload.data.sh" should be "upload_data.sh". I still get the "upload_data.sh: line 2: /cygdrive/c/Program Files (x86)/Google/google_appengineappcfg.py: No such file or directory" – user1832478 Mar 15 '14 at 00:59

score 1 · Answer 2 · answered Mar 15 '14 at 21:16

I finally got this working by running windows command prompt in administrator mode. I then navigated to google_appengine directory. I then used the following command in Windows cmd:

appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename places.csv --kind=Place -e mymail@mail.com

I also placed the the bulkloader.yaml and places.csv file in google_appengine directory before running the command. It will ask for the password simply press enter then the file should upload to the localhost.

swapsCAPS · Answer 3 · 2014-09-18T12:45:50.873

Ran into the same problem.
The tutorial is not very well written at this particular point. I did the following on an Ubuntu system

Download google_appengine for Python https://developers.google.com/appengine/downloads
Extract the folder somewhere
Copy the places.csv and bulkloader.yaml from MobileAssistant-Data to the folder
Create the file upload_data.sh file in that folder

Write the following two lines and save the file

#!/bin/sh 
./appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e nobody@nowhere.com

Make the file executable chmod a+x upload_data.sh

If all wen't well, running:

<google_appengine folder>./upload_data.sh places.csv Place

Should output something like:

12:55 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20140918.125544
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for nobody@nowhere.com: 
[INFO    ] Opening database: bulkloader-progress-20140918.125544.sql3
[INFO    ] Connecting to localhost:8888/remote_api
[INFO    ] Starting import; maximum 10 entities per post
.
[INFO    ] 2 entities total, 0 previously transferred
[INFO    ] 2 entities (1696 bytes) transferred in 58.8 seconds
[INFO    ] All entities successfully transferred

Hope it helps!

Google App Engine and Upload script

3 Answers3

Linked