2

I am trying to bootstrap a Centos7 EC2 instance (ami-02eac2c0129f6376b) with bash user-data. Because it runs as root and I need to create a lot of things as the centos user, I use sudo -Hu centos <cmd> many times. Each call introduces a 25 second delay. I have hundreds. What can I do to speed this up?

This is a known issue, but I cannot find any solutions.

I have tried:

  • Add and /etc/hosts entry for my hostname
printf "\n127.0.0.1   %s %s\n" "$(hostname -f)" "$(hostname -s)" | tee -a /etc/hosts
  • Remove myhostname from /etc/nsswitch.conf
sed -Ei 's/\s*myhostname//' /etc/nsswitch.conf

These delays are EXTREMELY painful because I am currently in the Trial and Error phase of building out new user-data scripts.

What can I do?

Bruno Bronosky
  • 4,529
  • 3
  • 26
  • 34
  • Do you not have DNS lookups and hostnames enabled for your VPC? – Michael - sqlbot Aug 02 '19 at 15:22
  • @Michael-sqlbot I curl several resources in my user-data. They have no problem doing lookups. But for some reason, every `sudo` takes 25 seconds to make the call to the command. – Bruno Bronosky Aug 02 '19 at 20:39
  • Hmmm... does `dig $(hostname)` work (return the instance's internal IP) and `dig -x z.z.z.z` (z.z.z.z = instance's private IP) work (return the hostname) or do they time out or return an error? IIRC sudo tries to look up the machine's IP... 25 sec seems like a very long time if this is the problem, but this is the only idea I have. – Michael - sqlbot Aug 02 '19 at 23:01

2 Answers2

2

While it's better to figure out why sudo is slow and resolve that, you can group your commands together to only call sudo once. Inside your script, you could do something like this:

sudo -Hu centos bash <<EOF
somecommand
somecommand2
morecommands
EOF
jordanm
  • 889
  • 5
  • 9
2

A bug in the SELinux policy included with RHEL and CentOS 7 caused sudo to wait for exactly 25 seconds on each command. This bug was fixed with the selinux-policy-3.13.1-229.el7_6.10 package. You should create a new AMI with this package (and preferably the whole system) updated.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • I am using - Name: selinux-policy Arch: noarch Version: 3.13.1 Release: 229.el7_6.15 – Bruno Bronosky Aug 04 '19 at 01:53
  • @BrunoBronosky Is that already in your AMI or does your cloud-init update to it on first boot? If it's already in your AMI then the problem may be elsewhere, and it's time to start looking at how your AMI was put together. But that doesn't appear to be the latest CentOS 7 AMI, so you probably should just go ahead and get the latest AMI. – Michael Hampton Aug 04 '19 at 01:57
  • My `yum update -y` updates selinux-policy.noarch 0:3.13.1-229.el7_6.6 to 0:3.13.1-229.el7_6.15 and selinux-policy-targeted.noarch 0:3.13.1-229.el7_6.6 to 0:3.13.1-229.el7_6.15 Do I need to do something after the `yum update` to get it to take effect? – Bruno Bronosky Aug 04 '19 at 02:01
  • @BrunoBronosky The update needs to be installed before the first run of cloud-init, so you need a new AMI. You should be on ami-089ccd342f0be98ab, or something later, not ami-02eac2c0129f6376b. If you aren't getting that AMI, see [this reddit thread](https://www.reddit.com/r/aws/comments/bkffft/centos_7_error_launching_source_instance/). – Michael Hampton Aug 04 '19 at 02:02
  • I cannot find that image. I have tried `aws ec2 describe-images --image-ids ami-089ccd342f0be98ab` and even `aws ec2 describe-images --owners aws-marketplace --filters '[ {"Name": "product-code", "Values": ["aw0evgkw8e5c1q413zgy5pjce"]} ]' --query 'sort_by(Images, &CreationDate)'` the image I am using is the latest I can find. Here is how I am actually selecting my image. https://stackoverflow.com/a/56401124/117471 – Bruno Bronosky Aug 04 '19 at 02:38
  • Interesting. I wonder if they pulled the newer image for some reason? You might be back to constructing your own AMI. – Michael Hampton Aug 04 '19 at 03:17
  • 1
    @BrunoBronosky CentOS 7.7 is being released now. Hopefully they will very shortly get you an updated AMI to work with, which should solve the problem. In any case the wait should not be too long. – Michael Hampton Sep 19 '19 at 20:49