2

I'm trying to build a mysql on microk8s. I get the following error when I connect to the nfs server or not. Although I have given the permissions of the file directories (chmod -R 777 .) on the nfs server and also on the local, I can't get rid of getting the error. I'm getting an error in mysql, which I installed and ran without any problems before. The image I am using = docker.io/bitnami/mysql:8.0.23-debian-10-r0

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  86s               default-scheduler  Successfully assigned gib/mysql-primary-0 to localhost.localdomain
  Normal   Pulled     85s               kubelet            Container image "docker.io/bitnami/mysql:8.0.23-debian-10-r0" already present on machine
  Normal   Created    85s               kubelet            Created container mysql
  Normal   Started    84s               kubelet            Started container mysql
  Warning  Unhealthy  7s (x5 over 46s)  kubelet            Readiness probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!

Describe image as above. When I enter the pod, there is no mysql.sock in this path. I get this error on both rocky linux and ubuntu 20.04. My conf file is as follows. /etc/my.cnf does not occur in the server.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: gib
data:
  
  my.cnf: |-
      [mysqld]
      default_authentication_plugin=mysql_native_password
      skip-name-resolve
      explicit_defaults_for_timestamp
      basedir=/opt/bitnami/mysql
      plugin_dir=/opt/bitnami/mysql/plugin
      port=3306
      socket=/opt/bitnami/mysql/tmp/mysql.sock
      tmpdir=/opt/bitnami/mysql/tmp
      max_allowed_packet=16M
      bind-address=0.0.0.0
      pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
      log-error=/opt/bitnami/mysql/logs/mysqld.log
      character-set-server=utf8
      collation-server=utf8_bin
      max_connections=650
      innodb_flush_log_at_trx_commit=0
      innodb_rollback_on_timeout = ON
      innodb_buffer_pool_size=512M
      [client]
      port=3306
      socket=/opt/bitnami/mysql/tmp/mysql.sock
      default-character-set=UTF8
      plugin_dir=/opt/bitnami/mysql/plugin
      [manager]
      port=3306
      socket=/opt/bitnami/mysql/tmp/mysql.sock
      pid-file=/opt/bitnami/mysql/tmp/mysqld.pid

The logs of my mysql pod are as follows. Then the pod restarted.

mysql 18:28:17.19
mysql 18:28:17.21 Welcome to the Bitnami mysql container
mysql 18:28:17.21 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mysql
mysql 18:28:17.21 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mysql/issues
mysql 18:28:17.22
mysql 18:28:17.22 INFO  ==> ** Starting MySQL setup **
mysql 18:28:17.24 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 18:28:17.29 INFO  ==> Initializing mysql database
mysql 18:28:17.37 WARN  ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 18:28:17.38 INFO  ==> Installing database
mysql 18:30:33.33 INFO  ==> Starting mysql in background
mysql 18:30:41.68 INFO  ==> Configuring authentication
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql 18:30:44.40 INFO  ==> Configuring replication in master node
mysql 18:30:45.04 INFO  ==> Running mysql_upgrade
mysql 18:30:45.04 INFO  ==> Stopping mysql
mysql 18:30:52.08 INFO  ==> Starting mysql in background

I couldn't find what the problem was, even though I've installed without problems before. Many people have encountered this error, but the solutions did not work for me. Can you help me to solve this error I got in mysql on microk8s?

sercanezelhan
  • 175
  • 6
  • Why would you use bitnami?! They're like the worst images for anything... From your `describe`, we can see mysql server isn't started (socket file missing): could you share the logs from your mysql container? Also: try to replace `localhost` by `127.0.0.1` in your liveness/readiness probes. – SYN Jun 19 '22 at 16:34
  • While using NFS for databases ... is a recipe for disaster. And any chmod 777 is another "you're doing it wrong" moment. Have you checked if using an emptyDir allows your container to start properly? Could be worth giving it a shot, make sure NFS isn't related to your issue. – SYN Jun 19 '22 at 16:41
  • Thank you for the answer. I updated the question at your request and added the logs as well. Can you suggest a way where I can find a helm or deployment that will run mysql in cluster mode and stable? – sercanezelhan Jun 19 '22 at 18:35
  • I would look into mysql official tooling, starting with their operator, which is capable of managing clusters. See docs: https://dev.mysql.com/doc/mysql-operator/en/ . Or: MariaDB's: https://mariadb.com/kb/en/kubernetes-operators-for-mariadb/. – SYN Jun 19 '22 at 19:39
  • While reading your logs: sounds like mysql is started. And yet, the liveness can't find the socket file. my.cnf configmap and liveness failure in pod description would both mention `opt/bitnami/mysql/tmp/mysql.sock` .... next guess is, the "Starting mysql in background" doesn't prove process did start: maybe it's stuck? I would enter that pod (kubectl exec -it pod-name -- /bin/sh), try to figure out what's it doing (processes running), and check the entrypoint script for what's supposed to show next. – SYN Jun 19 '22 at 19:47
  • and.... more generally, when thinking about introducing a stateful workload in Kubernetes, i would remind myself about some Google engineer, that once said: "Kubernetes supports stateful workload, I don't" -- https://twitter.com/kelseyhightower/status/963419099144495104?lang=en . Consider your options. Ask your colleagues. Does Kubernetes brings any value? Are you up to that task? While you may all be familiar with mysql clustering issues and how to fix those: what about containerized context? Anyone knows how to debug mysql process not starting in a container in crashloopbackoff? – SYN Jun 19 '22 at 19:56
  • ... still I don't mean to say that's a no-no. For a dev platform, it could be quite useful, allowing anyone in your team to spawn disposable clusters fast, .... For production, your team ability to operate those would be a crucial factor. While bare-metal/regular VM would usually offer better (more predictible) performances, less likely to suffer from network saturation, OOM caused by other workloads that would be poorly configured, ... If you want a cluster for your production application, hosting this in Kubernetes is a lot of work, trial & error, ... – SYN Jun 19 '22 at 20:04

0 Answers0