0

I have a CentOS 7.9 machine with AWS CLI installed on it. I can successfully run this command on its own: aws s3 sync s3://bucketname /data/bucketname/ but when I try to run it via crontab, it doesn't work. All I see in the /var/log/cron is that the command ran, but there is no data in that /data/bucketname directory. This is what my crontab looks like:

*/5 * * * * sh /root/script.sh

And this is what my script looks like:

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
SHELL=/bin/bash
/bin/aws s3 sync s3://bucketname /data/bucketname/

which aws returns /bin/aws so I tried to add that to my script.sh file but that did not make a difference. Any help would be much appreciated.

Tina
  • 21
  • 4
  • 2
    Does the root user have aws configuration in ~/.aws? – jordanm Mar 17 '22 at 14:28
  • 1
    What does `grep CRON /var/log/syslog` show? – Anon Coward Mar 17 '22 at 14:49
  • @jordanm - ~/.aws/config shows [default] region = regionname – Tina Mar 17 '22 at 15:04
  • @AnonCoward - /var/log/syslog no such file or directory – Tina Mar 17 '22 at 15:04
  • Looking at `~/.aws/config` is probably the not the root user's home directory unless you first sudoed into the root user account. – Mark B Mar 17 '22 at 15:29
  • @Mark, I sudo'ed and became root. – Tina Mar 17 '22 at 15:53
  • Maybe worth running aws with --debug option. – jarmod Mar 17 '22 at 16:34
  • Can you append `/var/log/awscron.log 2>&1` to your `cron` and share the logs? `(*/5 * * * * sh /root/script.sh >> /var/log/awscron.log 2>&1)` – Riz Mar 17 '22 at 16:48
  • @Riz - Thank you for the great reply. I added that line to my cron and this is what I see in the log: fatal error: could not connect to the endpoint URL: "https://s3.regionname.amazonaws.com/bucketname?prefix=&encoding-type=url" – Tina Mar 18 '22 at 15:57
  • @Riz - We have a different URL for aws, so the amazonaws.com part has to be changed. How can I do that? – Tina Mar 18 '22 at 16:23
  • Hi @Tina, when you say "~/.aws/config shows [default] region = regionname", What is the region name? Also check https://stackoverflow.com/questions/40409683/aws-s3-cli-could-not-connect-to-the-endpoint-url, you might have wrong region name in your config. Also, it doesn't depend on the url of your s3. It surely of that of the region. – Riz Mar 19 '22 at 21:49
  • @Riz The region name is us-east-1, I was able to resolve the issue by adding --endpoint-url and specify the very unique/customized url that we have for the console – Tina Mar 21 '22 at 18:08
  • Also, how do you prevent the aws sync command from re-adding the same files every time the cron runs? I now have duplicate data after each run – Tina Mar 21 '22 at 18:39
  • @Tina, I have never come across such issue of duplication in case of s3 sync. Is it possible that you have versioning on and in the console you have versioning enabled that's why you see duplicates? – Riz Mar 21 '22 at 22:09
  • @Riz I have already checked, versioning is disabled – Tina Mar 22 '22 at 18:37
  • It's strange that it duplicates the files. You can do `aws s3 sync s3://bucketname /data/bucketname/ --dryrun` and update the question maybe or open another one with more information/details? There must be some difference somewhere. Does the path/prefix (in s3) change when you sync from s3 to your pc? – Riz Mar 22 '22 at 21:40
  • 1
    @Riz, i am actually trying to sync from a S3 bucket into a directory on an EC2 instance. Will try to see maybe new data was added to the original bucket and report back – Tina Mar 23 '22 at 12:10
  • @Riz I am showing the actual bucket started to get more files added to it, so we're good. Thanks again for your help – Tina Mar 24 '22 at 16:55
  • @Tina, perfect! No worries. – Riz Mar 24 '22 at 16:58
  • @Riz I would like to have 7 of these sync commands in my script, they are very long so I have to add line breaks. Is this a correct format? `code /bin/aws s3 sync s3://bucketname/subdir1 /data/subdir1/ --endpoint-url ... --no-verify-ssl && \ /bin/aws s3 sync s3://bucketname/subdir2 /data/subdir2/ --endpoint-url ... --no-verify-ssl && \ /bin/aws s3 sync s3://bucketname/subdir3 /data/subdir3/ --endpoint-url ... --no-verify-ssl `code After each \ a new line starts, but somehow i can't add a br in this comment – Tina Apr 19 '22 at 17:14
  • Hello @Tina, yes, the format is totally fine. You can do a test with `--dryrun` as well to be 100% sure. I don't know much about your needs but you can have a look at `--include` and `--exclude` options as well. Maybe you can simplify your script. Also if all these sync commands are not dependent on each other, I would suggest to avoid `&&` as if one `sync` fails, the script won't execute the other remaining `sync`s. – Riz Apr 19 '22 at 22:14
  • @Riz Thanks much for your reply. So I should just have each sync command in a separate line with no line break and no && since they are not dependent on each other, this would be my sync.sh script. Somethings like this: aws sync command 1 Then a new line for each aws sync command? – Tina Apr 20 '22 at 13:34
  • @Tina, yes exactly. This way when one command finishes, it will go to the next line and execute next command(sync). You can improve it to do all commands in parallel. You can do `aws sync command 1 &` the new line then new sync with & at the end of each command. This single & put your sync command in background and hops to the next command instead of waiting for the first to complete. – Riz Apr 20 '22 at 13:45
  • 1
    @Riz Thanks much! This is a brilliant idea. I appreciate your help – Tina Apr 21 '22 at 12:16

0 Answers0