16

Last week I got a call from a scared customer because he thought his website was hacked. When I looked up his website I saw the apache2 default page. That night my server (Ubuntu 16.04 LTS) had upgraded and rebooted. Normally when something goes wrong I would've got alerted during the night. This time not, because the monitoring system checks for HTTP status code 200, and the apache2 default page comes with status code 200.

What happened is that during startup apache2 was faster to bind to port 80 and 443 than my actual webserver nginx. I did not install apache2 myself. Through aptitude why apache2 I found out the php7.0 package requires it.

Simply removing apache2 won't work because apparently php7.0 requires it. Is it somehow possible create a restriction so that only nginx is allowed to bind to port 80 and 443?

Other solutions are more than welcome too.

techraf
  • 4,243
  • 8
  • 29
  • 44
Boyd
  • 271
  • 2
  • 5
  • 15
    And this is why you should configure your live servers to only update when you explicitly request an update, so you can first test your updates on a development machine. – Nzall Apr 13 '17 at 09:27
  • 2
    I do not test the upgrades on a test machine first, but do always check the changelogs before **manually** scheduling the upgrade. Also it looks like apache2 slipped through during an earlier upgrade. Its just that this time it rebooted apache2 was the first to bind to the http and https ports. – Boyd Apr 13 '17 at 09:30
  • 9
    As a side note - `This time not, because the monitoring system checks for HTTP status code 200`. You can improve the monitoring system by making it to check actual content of the web page (some particular string in the body or header), this will be more reliable. – VL-80 Apr 13 '17 at 12:39
  • @Boyd Thank you, and as a matter of fact this has already been implemented since yesterday :) – Boyd Apr 13 '17 at 12:45
  • 2
    @Boyd *I do not test the upgrades on a test machine first, but do always check the changelogs* But you've just experienced just how unreliable that method is. Reading a changelog can't tell you what the impact on a deployed system will be, nor will it tell you about bugs or incompatibilities that are introduced. – Andrew Henle Apr 13 '17 at 13:39
  • 2
    @AndrewHenle Let it be clear that I'm not trying to say testing the actual upgrade is unnecessary. It's just the current situation I have to deal with. Until we can actually move to a setup where we have a replica of the production server available I still highly prefer weekly upgrades over not upgrading until we can test the upgrades first. It probably means I will be woken up by the monitor some times though. – Boyd Apr 13 '17 at 13:53
  • 5
    @Nzall to be fair it sounds like this sort of issue might not have showed up on a test machine... he effectively has a race condition as to whether apache2 or nginx will bind the ports, and the test machine could theoretically end up having nginx win (just by chance) for the duration of testing so the issue wouldn't be discovered. – Doktor J Apr 13 '17 at 23:49
  • One thing you've learned here is that checking for a status code is not enough. You should have a specific static web page and verify its checksum. Also, you should check which services are enabled after each upgrade. Using a configuration management system should help. And get a virtualbox or vmware box to run your test env in. (Everyone has a test environment, smart people have one that's not also production...) – Jenny D Apr 14 '17 at 11:02
  • 1
    @Boyd: You should create a new question with the real problem: How to install nginx and PHP on Ubuntu without installing Apache. There is already a similar question on Stack Overflow, but its title is unclear: [Ubuntu Server Installing PHP 7 WITHOUT Apache](http://stackoverflow.com/questions/34880267/ubuntu-server-installing-php-7-without-apache). – David Cullen Apr 14 '17 at 14:30
  • "php7.0 package requires it." - TRWTF, right there. – Kevin Apr 14 '17 at 23:18
  • @Kevin it doesn't. It's a metapackage that depends on any of three PHP SAPIs, the default of which is the Apache module, which of course depends on Apache. – muru Apr 17 '17 at 03:50

5 Answers5

29

You can't prevent a port from being bound by the wrong service. In your case, just remove apache from autostart and you should be good.

For 16.04 and newer:

sudo systemctl disable apache2

For older Ubuntu versions:

sudo update-rc.d apache2 disable
Gerald Schneider
  • 23,274
  • 8
  • 57
  • 89
  • 2
    I will do this, but I hoped I could defend myself from future situations in which another package that binds to port 80 and 443 is unknowingly installed on my system as a dependency of another package. – Boyd Apr 13 '17 at 07:37
  • 2
    Since this is 16.04, also: `systemctl disable apache2` – muru Apr 13 '17 at 09:26
  • 12
    @Boyd: Why are you blindly installing packages "unknowingly"? How come, on your live server used by customers, you are not even reading which packages and dependencies are being installed? And how come you're not testing everything on a mirror server before executing? These are _basic operations principles_ and will solve all of your problems. – Lightness Races in Orbit Apr 13 '17 at 11:35
  • 6
    @BoundaryImposition Wanting to defend against software binding to ports doesn't mean I'm just blindly installing packages. But we're people too, mistakes are made. Unfortunately we're not able to test every operation on a dummy server first but in this case it would not have immediately showed the issue, because apache2 was installed a week before the issue appeared (the system was even rebooted in the meantime without any issues). While not being able to test every upgrade we still upgrade weekly. We prefer up-to-date security patches over the limited (through monitoring) downtime risk. – Boyd Apr 13 '17 at 12:08
  • 3
    @Boyd: _"Unfortunately we're not able to test every operation on a dummy server first"_ Why not? Are your customers aware that you skip this procedure? – Lightness Races in Orbit Apr 13 '17 at 12:25
27

If you really aren't using apache2, and it's PHP 7.0 that's requiring it, then it looks like you have libapache2-mod-php7.0 installed. That package is useless without Apache. Since you're using nginx, you likely also have php7.0-fpm or php7.0-cgi installed, either of which is sufficient for satisfying php7.0's dependency requirements:

$ apt-cache depends php7.0
php7.0
 |Depends: php7.0-fpm
 |Depends: libapache2-mod-php7.0
  Depends: php7.0-cgi
  Depends: php7.0-common
  Conflicts: <php5>

If you do have either of php7.0-{fpm,cgi} installed, you can go ahead and uninstall Apache.

muru
  • 589
  • 8
  • 26
  • 6
    I indeed learned today that in my situation I'm better off with just installing `php7.0-fpm` and not the `php7.0` package. This is also advised by Ondřej Surý https://github.com/oerdnj/deb.sury.org/wiki/PPA-migration-to-ppa:ondrej-php#why-does-installing-php71-also-install-libapache2-mod-php71 – Boyd Apr 13 '17 at 09:37
  • 5
    This is the real solution to the real problem: How to install nginx and PHP on Ubuntu without installing Apache. – David Cullen Apr 14 '17 at 14:33
2

To answer your question, you can probably restrict a port to a specific application by using SElinux. I haven't used it myself and have only superficial knowledge of its capabilities, but here is a pointer I found in this site:

https://serverfault.com/a/257056/392230

In that answer, wzzrd seems to show how to give a specific application (foo) permission to bind to a specific port (803). You'd just have to have the policy set-up so that only your application (nginx) is allowed the ports you specify (80 and 443).

Basing myself on wzzrd's answer, it might be as simple as adding this to the policy

allow nginx_t nginx_port_t:tcp_socket name_bind;

and running this

semanage port -a -t nginx_port_t -p tcp 80
semanage port -a -t nginx_port_t -p tcp 443

Though, I imagine you'll also need a line in the policy that specifies that no other program may bind to those ports.

In the end, I'm just guessing what the appropriate configuration is.

Anyway, I don't think there's been an Ubuntu that has SElinux installed and enabled by default. Because I believe it requires applying certain patches to various utilities and a kernel option, it might be easier to simply use Centos which does have SElinux installed and enabled from the get-go.

Sorry, I'm not of more help. Maybe some other time, I'll download an image of Centos and try this; it'll be a good learning step. I'll update this answer if I do.

JoL
  • 121
  • 4
2

Something that I haven't seen in the answers yet, but is still a possibility:

Change the Apache config to listen to another port, just in case. You can do that by opening the Apache config file, and changing the lines that have Listen 80 to another port.

Milan
  • 21
  • 1
  • That "solves" the issue just the same as the accepted answer, but with the added issue of then having to explain/document your change. Also, while it does solve a problem, neither solves the entire problem. If apache is disabled but next time it reboots, application X binds to port 80, you have the same fault again. – Darren H Apr 14 '17 at 14:02
0

I don't have an answer to your exact question, but maybe you need to look at your distro. I would consider any distro that enables services (apache2 here) on install to be insecure. Consider looking into a distro that doesn't do that. I can't say I've ever seen that behavior on Archlinux, I'm sure there are others.

phelbore
  • 25
  • 2
  • 1
    So what do you recommend, to format the server and install some other distribution? And why do you believe that ubuntu is not able to handle specific services, as some other answer indicated? This answer is a religious comment and not helpful whatsoever. – Wtower Apr 14 '17 at 08:30
  • I wouldn't format the server right now and install a new distribution, but I'd certainly change next time I had to upgrade servers. Ubuntu has just been shown in this post to be unsuitable since it's enabling services that haven't been configured (exceptions to this would be something like a graphical login or sound service, things that normally just work and aren't exposed to the public internet). I was afraid the answer might come off a little religious, but that wasn't the intent, it was trying to point out a solution to what I saw as the larger problem. – phelbore Apr 14 '17 at 14:57
  • 1
    Though I agree with you that it's improper of a distro to do that, I think this would've been more appropriate as a comment, as it doesn't really answer the question. – JoL Apr 15 '17 at 07:36
  • It's a fair point. – Lightness Races in Orbit Apr 15 '17 at 14:15