0

I'm working on setting up a MariaDB 3 Node Cluster and using Maxscale as the proxy. I had set up a practice config on some local KVM machines, worked without a hitch. So I went to spin up the production servers and I'm getting an error I can't make sense of. If I run any command in maxctrl at all it throws the same error:

ERROR
The requested URL could not be retrieved
The following error was encountered while trying to retrieve the URL: http://localhost:8989/v1/maxscale/modules/mariadbmon/
Connection to ::1 failed.
The system returned: (99) Cannot assign requested address
The remote host or network may be down. Please try the request again.

Ok so it sounds like something was using port 8989 before Maxscale, let's check with lsof -i -P -n | grep 89:

maxscale 1117 maxscale   23u  IPv4  19765      0t0  TCP 127.0.0.1:8989 (LISTEN)

SELinux is set to Permissive for testing, Firewalld is off for testing.

Someone suggested it might be an IPv6 issue since it says connection to ::1 but I can't see what the difference would be between my test and pro machines as they both have the same default loopback adapter settings in lo and both have the same aliases in /etc/hosts

Suggestions for debugging?

EDIT: Trying out a few of the recommendations from markusjm below: 1) Nothing in logs that jumps out at me, here's everything up until the listener claims to be started:

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Sun Feb  2 21:31:23 2020
----------------------------------------------------------------------------
2020-02-02 21:31:23   notice : syslog logging is enabled.
2020-02-02 21:31:23   notice : maxlog logging is enabled.
2020-02-02 21:31:23   notice : Using up to 3.51GiB of memory for query classifier cache
2020-02-02 21:31:23   notice : Working directory: /var/log/maxscale
2020-02-02 21:31:23   notice : The collection of SQLite memory allocation statistics turned off.
2020-02-02 21:31:23   notice : Threading mode of SQLite set to Multi-thread.
2020-02-02 21:31:23   notice : MariaDB MaxScale 2.4.5 started (Commit: 61b8bbf7f63c38ca9c408674e66f3627a0b2192e)
2020-02-02 21:31:23   notice : MaxScale is running in process 8036
2020-02-02 21:31:23   notice : Configuration file: /etc/maxscale.cnf
2020-02-02 21:31:23   notice : Log directory: /var/log/maxscale
2020-02-02 21:31:23   notice : Data directory: /var/lib/maxscale
2020-02-02 21:31:23   notice : Module directory: /usr/lib64/maxscale
2020-02-02 21:31:23   notice : Service cache: /var/cache/maxscale
2020-02-02 21:31:23   notice : Worker message queue size: 1.00MiB
2020-02-02 21:31:23   notice : No query classifier specified, using default 'qc_sqlite'.
2020-02-02 21:31:23   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
2020-02-02 21:31:23   notice : Query classification results are cached and reused. Memory used per thread: 449.02MiB
2020-02-02 21:31:23   notice : The systemd watchdog is Enabled. Internal timeout = 30s
2020-02-02 21:31:23   notice : Loading /etc/maxscale.cnf.
2020-02-02 21:31:23   notice : /etc/maxscale.cnf.d does not exist, not reading.
2020-02-02 21:31:23   notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
2020-02-02 21:31:23   notice : [readwritesplit] Initializing statement-based read/write split router module.
2020-02-02 21:31:23   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
2020-02-02 21:31:23   notice : [readconnroute] Initialise readconnroute router module.
2020-02-02 21:31:23   notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
2020-02-02 21:31:23   notice : [mariadbmon] Initialise the MariaDB Monitor module.
2020-02-02 21:31:23   notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so
2020-02-02 21:31:23   notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
2020-02-02 21:31:23   notice : Loaded module mariadbbackendauth: V1.0.0 from /usr/lib64/maxscale/libmariadbbackendauth.so
2020-02-02 21:31:23   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
2020-02-02 21:31:23   notice : Loaded module mariadbauth: V1.1.0 from /usr/lib64/maxscale/libmariadbauth.so
2020-02-02 21:31:23   notice : Started REST API on [127.0.0.1]:8989
2020-02-02 21:31:23   notice : MaxScale started with 8 worker threads, each with a stack size of 8388608 bytes.
2020-02-02 21:31:23   notice : Starting a total of 2 services...
2020-02-02 21:31:23   notice : Server 'server1' version: 10.3.21-MariaDB-log
2020-02-02 21:31:23   notice : Server 'server2' version: 10.3.21-MariaDB-log

2) curl localhost:8989/v1/maxscale returns the 99 error code as above. If I do curl 127.0.0.1:8989/v1/maxscale it returns a different 111 error.

<blockquote id="error">
<p><b>Connection to 127.0.0.1 failed.</b></p>
</blockquote>

<p id="sysmsg">The system returned: <i>(111) Connection refused</i></p>

3) tcpdump shows that absolutely nothing is coming across the wire, which is really weird. I tried tcpdump -v -i ens192 'port 8989' and tcpdump -v -i lo 'port 8989' and then both curl methods as above, and get the same result:

tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Aaron Chamberlain
  • 381
  • 1
  • 3
  • 13

2 Answers2

1

MaxCtrl uses the MaxScale REST API to execute commands. Most of the time error 99 appears on the client side when the system is incapable of creating more TCP sockets. This type of an error should be transient so it should clear up over time. Inspecting the number of TCP sockets and their state should indicate whether this is the case.

The usual Maxscale REST API debugging steps are:

  • Check the MaxScale error logs at /var/log/maxscale/maxscale.log and verify that it started successfully listening on the correct port.
  • Test that HTTP connections work by using a different client e.g. curl localhost:8989/v1/maxscale
  • Capture the network traffic with tcpdump -v -i lo 'port 8989' and inspect it for any clues

If none of these steps resolve the problem, you can always open a bug report on the MariaDB Jira under the MaxScale project.

Aaron Chamberlain
  • 381
  • 1
  • 3
  • 13
markusjm
  • 328
  • 1
  • 6
0

My system was set up to use an HTTP Proxy, and the proxy was not allowing the connection on port 8989.

In /etc/environment I had:

http_proxy=http://<lan_ip>:3128
https_proxy=http://<lan_ip:3128

When I removed these, closed the SSH session, and came back in commands like maxctrl list servers now work. Will have to work on a resolution around the proxy.

Aaron Chamberlain
  • 381
  • 1
  • 3
  • 13