6

I have a batch file C:\upload_to_s3.bat. In this file, there is a line:

aws s3 sync D:\S3\batch1\ s3://MyBucket/batch1 --exclude *.bat

I have Windows task scheduler "S3 Hourly Sync" that runs every hour to trigger to run C:\upload_to_s3.bat. But this command does not do anything - the file upload never happened.

It runs perfectly if I double click on C:\upload_to_s3.bat.

This is Windows 2008 Standard server. I have installed AWS CLI and configured with the command "aws configure", and entered my access key and secret key. That is why it runs if I double click the batch file.

The Windows Task Scheduler is run as "[MyServer]/Administrator" account.

So why doesn't the "aws s3 sync" command work when triggered by the task scheduler?

Do I need to add a line before "aws s3 sync" to set credentials first? If so, how?

Thank you!

Graviton
  • 81,782
  • 146
  • 424
  • 602
user3703038
  • 61
  • 1
  • 1
  • 2
  • 1
    Were you logged in as [MyServer]\Administrator when you ran `aws configure`? The `aws configure` command is documented to use the config file at `~/.aws/config` (on Windows, I believe this maps to "%USERPROFILE%\.aws\config"). Does that file exist for the batch file user? – James Jun 03 '14 at 23:51
  • yes, I logged in as Administrator and I can see the file C:\Users\Administrator\.aws\config. Any other thoughts? Please feel free to ask questions - one of them may just sparkle. – user3703038 Jun 04 '14 at 04:15
  • Was there any output from the batch file? If not, can you capture the output to file? You might run another command, maybe to list a file from S3, that you can run to check the permission situation safely. – James Jun 04 '14 at 05:07
  • I modify the new line to "aws s3 sync D:\S3\batch1\ s3://MyBucket/batch1 --exclude *.bat > D:\logs\s3.txt" The output file is empty - actually the output file was not even created. The commands before and after this line were executed - that means the task scheduler works fine. The "aws s3" line is the only problem. More thoughts? – user3703038 Jun 04 '14 at 14:50
  • Maybe user account control is preventing local file access? I recommend trying to separate local issues from AWS issues, and I would try this by adding two additional lines to the batch file to capture text output - 1) `type "%USERPROFILE%\.aws\config"` to ensure the batch file can read the credentials, and 2) `aws s3 ls` as a minimal command that uses AWS creds without doing anything. Both should generate output, error or not. – James Jun 04 '14 at 16:39
  • ok. I got the two lines here: 1) Unable to locate credentials. 2) Completed 1 part(s) with ... file(s) remaining. In my batch file, the order of commands are: 1) type ... 2) aws s3 sync... 3) aws s3 ls... So "Unable to locate credentials" is probably from the command "aws s3 sync". How do I resolve this? By the way, if I double click on the batch file, every line runs fine without errors (only the file names that are uploaded). – user3703038 Jun 04 '14 at 21:00
  • I am trying to add environment variables for the access key and secret key to see how it goes. I will come back tomorrow morning to update this. Thank you! – user3703038 Jun 04 '14 at 22:05
  • ok, it doesn't work. Adding system environment variables doesn't help. I am still searching for solutions. – user3703038 Jun 05 '14 at 12:17
  • I have fixed this issue by 1) copying the .aws/config under Administrator to All Users and Default User. 2) Adding environment variables to point to the config file. 3) Rebooting the server. Not all three may be necessary but it works now. – user3703038 Jun 05 '14 at 20:01

6 Answers6

8

i was able to solve the problem by adding the following 3 lines in the batch file BEFORE your s3 command

aws configure set AWS_ACCESS_KEY_ID <your acess key id>
aws configure set AWS_SECRET_ACCESS_KEY <your secret access key here>
aws configure set default.region eu-west-1

replace the stuff between angled brackets with your data and the region with your bucket's region. and yes ,there is no equal sign between "AWS_ACCESS_KEY_ID" and the key.

EKanadily
  • 3,831
  • 3
  • 35
  • 33
  • 1
    You don't need this if you set it up per AWS CLI documentation. CMD Prompt: `aws configure` on user running task scheduler (usually 'Administrator'), it should automatically create `c:\users\administrator\.aws\config` and `c:\users\administrator\.aws\credentials` files. You're set. Note: you can also setup different profiles as well. ( http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration ) – B. Shea May 20 '17 at 14:04
  • 1
    This should be avoided as it will display AWS credentials directly in the script. Using the shared credentials file created via `aws configure` is a better approach – scrthq Jan 03 '18 at 15:19
6

The reason this is happening, is because aws command is not finding the the config or the credentials file! And it happens for Win2012 too.

The problem is that Windows Task Schedule sets the %USERPROFILE% to C:\Users\Default instead of the actual user the task is running under!! ??? And AWS is using that to find the config and credentials files.

So the fix is to put something like this in your script (either BAT or PowerShell)

:: in a BAT script
set userprofile=C:\Users\%username%

# in a PowerShell
$env:USERPROFILE = "c:\users\$env:USERNAME"

Even the %homepath% variable is blank when running from a Scheduled Task.

I just discovered this after procrastinating trying to figure out why this was happening for us.

set AWS_DEFAULT_PROFILE=namedprofile
set AWS_CONFIG_FILE=C:\Users\xxxx\.aws\config

Only until I added the AWS_CONFIG_FILE line above in my script was AWS able to actually see the config file! However it can not find the credentials now, and I don't see anywhere how to set that.

This is the error I get without telling where the config file is:

Traceback (most recent call last):
  File "aws", line 27, in <module>
  File "aws", line 23, in main
  File "awscli\clidriver.pyc", line 50, in main
  File "awscli\clidriver.pyc", line 176, in main
  File "awscli\clidriver.pyc", line 157, in _create_parser
  File "awscli\clidriver.pyc", line 91, in _get_command_table
  File "awscli\clidriver.pyc", line 111, in _build_command_table
  File "botocore\session.pyc", line 680, in emit
  File "botocore\hooks.pyc", line 226, in emit
  File "botocore\hooks.pyc", line 209, in _emit
  File "awscli\customizations\preview.pyc", line 71, in mark_as_preview
  File "awscli\clidriver.pyc", line 349, in service_model
  File "awscli\clidriver.pyc", line 366, in _get_service_model
  File "botocore\session.pyc", line 256, in get_config_variable
  File "botocore\session.pyc", line 277, in _found_in_config_file
  File "botocore\session.pyc", line 349, in get_scoped_config
botocore.exceptions.ProfileNotFound: The config profile (backup) could not be found

And then this is the error I get after setting the AWS_CONFIG_FILE variable:

Unable to locate credentials
Completed 1 part(s) with ... file(s) remaining
user2415376
  • 3,440
  • 2
  • 16
  • 13
1

Include the following lines in your batch file:

set AWS_ACCESS_KEY_ID=your_access_key_id

set AWS_SECRET_ACCESS_KEY=your_secret_access_key

AWS Documentation

Hein Andre Grønnestad
  • 6,885
  • 2
  • 31
  • 43
user3815183
  • 21
  • 1
1

In windows bat file what works is a combination of user2415376 and docesam answers.

Here it is:

set userprofile=C:\Users\%username%
aws configure set AWS_ACCESS_KEY_ID <your_key>
aws configure set AWS_SECRET_ACCESS_KEY <your_secret>
aws configure set default.region <your_region>

other aws commands here
Auresco82
  • 603
  • 6
  • 14
0

Alternatively, we can copy the .aws folder to the users account folder (C:\Users\ %username%) that Task Scheduler uses to run the tasks with the AWS CLI scripts. The .aws folder contains the config and credential files, created with the 'aws configure' command.

Aprendiz
  • 23
  • 5
0
  1. On the Actions tab, set the action program to c:\windows\system32\cmd.exe with arguments to /C C:\upload_to_s3.bat The /C tells cmd to close after command execution. CMD.EXE will load the environment for the user in a command shell, and then run the batch file.
  2. On the General tab, make sure it is set up to run as the user that is set up to run aws. You mentioned you checked this. I mention it here as a reminder, if one is going to be using a different account. Also, make sure that account has the "Log On as a Service" right.
ericp
  • 611
  • 2
  • 11
  • 17