12

We have an application that has been running for about 2 years now without any issue.

This morning when I came in the site was getting an error:

Session: connection failed

I checked the database connections, checked the user, checked the grants for the user, everything looked fine.

I created a test page using the connection information in the .inc.xml file

Got a mysql connection error.

This is the original file:

   <TYPE>mysql</TYPE>
        <HOST>dbl</HOST>
        <USER>dbuser</USER>
        <PASSWORD key="PUT A KEY HERE TO DECRYPT THE PASSWORD">password</PASSWORD>

The hostname of the server is not dbl

I changed the second line to this:

   <TYPE>mysql</TYPE>
        <HOST>localhost</HOST>
        <USER>dbuser</USER>
        <PASSWORD key="PUT A KEY HERE TO DECRYPT THE PASSWORD">password</PASSWORD>

It connected fine.

Made the same change to the original .inc.xml file and the application was back up and running fine.

My question(s):

I checked the audit logs, connections, queries, the system has not been access for about 3 weeks.

The .inc.xml was last modified over a year ago.

Why would the application all of a sudden stop taking dbl as a host?

Where did that dbl hostname come from?

System Layout:

[root@acpr-web-x ~]# cat /proc/version
Linux version 2.6.32-358.2.1.el6.x86_64 (mockbuild@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Feb 20 12:17:37 EST 2013

[root@acpr-web-x ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Aug 15 2014 03:02:07

[root@acpr-web-x ~]# php -v
PHP 5.3.3 (cli) (built: Oct 23 2014 06:58:46)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

Mysql Server version: 5.1.73-log Source distribution
Anthony Fornito
  • 9,546
  • 1
  • 34
  • 124

1 Answers1

11

My first guess would be /etc/hosts which can override DNS (if it exists for the host) and substitute almost entirely for forward DNS when it's missing. For whatever reason this host (dbl) was in /etc/hosts and then it wasn't.

Secondarily, let's say /etc/resolv.conf lost the search option for the local domain, so that dbl.foo.com resolves but dbl does not... also a possibility.

Jason Chodakowski
  • 144
  • 1
  • 2
  • After checking the /etc/hosts file it was last modified 1/29/2015 and the dbl entry is still there pointed to the correct IP, /etc/resolv.conf last modified 11/8/2012 and looks correct. – Anthony Fornito Jun 20 '17 at 18:18
  • Ok then... how about this? From inside MySQL: `use mysql; select user,host,password from user` This may reveal a change in the host that the user 'dbuser' is allowed to access the db from. The potential causes are finite, should be able to determine this. I'd throw out a quick check on /etc/nsswitch.conf but that file is kind of obscure these days and so the chances of it changing are low, but it's worth a check (it controls the order in which things resolve on the system). – Jason Chodakowski Jun 20 '17 at 18:26