0

Problem

I have the following batch file (see below), on my own computer it runs successfully. It runs a Python executable compiled with GooeyParser, passes some variables to the batch script through temporary text files, then uses Google Dev Console to upload the file into GSC then BigQuery.

On my co-workers computer, the batch script just stops running after the gsutil command. There are no error messages, the script just stops running.

The individual commands run successfully on his computer.

Does anyone know why the batch script might only partially run?

Batch File

::spawns as a subprocess to avoid console window closing and losing error messages
if not defined in_subprocess (cmd /k set in_subprocess=y ^& %0 %*) & exit )

::python script writes out file location to temporary file
"BQ Single File Upload - dont run by itself.exe"
SET /p FILEUPLOAD=<temp_file_path
SET /p FILENAME=<temp_filename
SET /p DATASET=<temp_dataset
SET /p TABLE=<temp_table
SET /p SCHEMA=<temp_schema
DEL temp_file_path
DEL temp_filename
DEL temp_dataset
DEL temp_table
DEL temp_schema

gsutil cp %FILEUPLOAD% "gs://our_data/%FILENAME%"

cmd.exe /c bq mk %DATASET%

bq load --max_bad_records=5 --skip_leading_rows=1 --allow_quoted_newlines --schema %SCHEMA% %DATASET%%TABLE% "gs://our_data/%FILENAME%"
PAUSE
Mofi
  • 46,139
  • 17
  • 80
  • 143
Dominic Woodman
  • 719
  • 2
  • 8
  • 18
  • Why do you keep referring to it as a **BASH** script? – Squashman Nov 29 '16 at 18:58
  • Just looking at other questions about compiling python scripts there is probably some DLL on your computer that does not exist on the other computer. – Squashman Nov 29 '16 at 19:06
  • 2
    Do you run the batch file from within a command prompt window to avoid an automatic close of command process to be able to see error messages at all or do you just double click on the batch file which results in automatic close if Windows command processor stops execution of the batch file? – Mofi Nov 30 '16 at 06:22
  • 2
    Maybe it helps to enclose everything in quotes which perhaps contain a space, for example `gsutil cp "%FILEUPLOAD%" "gs://our_data/%FILENAME%"`. It would be also good to specify each console application to run with file extension and if possible also with full path. For example if `gsutil` is an executable with file extension `.exe`, use in the batch file `gsutil.exe`. Otherwise it could happen that there is a `gsutil.bat` and this file is found first by command processor which results in executing this batch file with no return to your batch file because of not using command `call`. – Mofi Nov 30 '16 at 06:40
  • @Squashman Ok so the gooey executable (i.e. the compiled python script) runs correctly by itself on his computer, so I don't think its that – Dominic Woodman Nov 30 '16 at 16:51
  • @Mofi Ok so adding "call" in front the functions worked. Previously the gsutil command would run successfully then, the window would stay open but no other commands would run. (If you swapped the order of gsutil & bq the same thing happened). Now after adding call, it runs through all the commands successfully. I'll be honest I'm not entirely sure why. – Dominic Woodman Nov 30 '16 at 17:04

1 Answers1

1

Adding call, (thank you @Mofi). Here is the final script, which now works on both our computers.

::spawns as a subprocess to avoid console window closing and losing error messages
if not defined in_subprocess (cmd /k set in_subprocess=y ^& %0 %*) & exit )

::python script writes out file location to temporary file
"BQ Single File Upload - dont run by itself.exe"
SET /p FILEUPLOAD=<temp_file_path
SET /p FILENAME=<temp_filename
SET /p DATASET=<temp_dataset
SET /p TABLE=<temp_table
SET /p SCHEMA=<temp_schema
DEL temp_file_path
DEL temp_filename
DEL temp_dataset
DEL temp_table
DEL temp_schema

call gsutil cp %FILEUPLOAD% "gs://our_data/%FILENAME%"

call cmd.exe /c bq mk %DATASET%

call bq load --max_bad_records=5 --skip_leading_rows=1 --allow_quoted_newlines --schema %SCHEMA% %DATASET%%TABLE% "gs://our_data/%FILENAME%"
PAUSE
Dominic Woodman
  • 719
  • 2
  • 8
  • 18