How to diagnose crash loops
Here's the steps I would suggest following. If you can provide the output of the logs in each step that'll help us see the issue.
Step 1: Install the OC client
There is only so far you can go through the UI, you'll need the oc client to get into deeper troubleshooting.
First, log into your cluster through the web interface. Choose the question mark at the top right of the screen and select 'Command Line Tools':
Follow the instructions to download and install the client.
Step 2: Show the current pods
Once you've got the client and logged in, run:
oc get pods
Which should show a list of the pod names. Please paste the content of the output in your question (as text, not an image!)
Step 3: Find the crashing pod and get its logs
You'll have a pod which is crashing, which will be called something like mysql-2-6c009. We'll need the logs from it. Paste the output of:
oc logs mysql-2-6c009
Step 4: If you cannot find the pod, redeploy it
If you cannot see the pod any more because the deployment has failed, try running:
oc rollout latest mysql
Then run oc get pods again until you see the crashing pod.
Step 5: Solving the problem!
The logs show the issue, you are not specifying the environment variables required to set the database up properly. We can see the same if we check the docs on ]OpenShift - MySQL](https://docs.openshift.com/enterprise/3.0/using_images/db_images/mysql.html#environment-variables):
You must specify the user name, password, and database name. If you do not specify all three, the pod will fail to start and OpenShift will continuously try to restart it.
To set the values, try this:
oc set env dc/mysql MYSQL_USER=user MYSQL_PASSWORD=P@ssw0rd MYSQL_DATABASE=db1
This will update your deployment config with the variables. It should re-deploy automatically if you have configured it to update when the configuration changes, if it doesn't, run:
oc rollout latest
In the future, you can create the app with the environment variables set in the first place like this:
oc new-app -e \
MYSQL_USER=<username>,MYSQL_PASSWORD=<password>,MYSQL_DATABASE=<database_name> \
registry.access.redhat.com/openshift3/mysql-55-rhel7
See these docs for details.
The logs show:
Tips and tricks
How pod names work
The pods have names which give some detail. Here's how they work:
mysql-2-deploy
This means it is the second deployment of the msql service. This is the pod which is orchestrating your specific deployment.
mysql-2-6c009
This means it is the mysql service, deployed during the second deployment. The random six digits at the end come from the pod id, they have to be there because you could deploy many instances of a service to many pods.
Looking at pods
As you get more familiar with the commandline tool, you might find yourself running oc get pods and similar commands a lot. If you are linux, you can use the watch tool to help (on Mac, just do brew install watch). Then run:
watch -n 1 -d oc get pods
This command will show you a live view of the pods, updated every second:
watch # run the following command repeatedly, showing the output
-n 1 # run every second (this is optional, the default is 2s)
-d # show a diff, highlighting the changes as they happen
oc get logs # the command to watch
This command is super useful and you'll use it a lot!
Quickly get logs for a pod
Try this bash function:
function podlogs() {
echo "Getting logs for $1 for the last $2 duration"
oc logs -f --since=$2 `oc get pods | grep $1 | grep 'Running' | grep -Ev 'deploy' | awk '{print $1}'`
}
It'll let you run a command like this:
# get all logs for containers which match 'mysql' for the last 5 mins
podlogs mysql 5m
Kudos to my buddy Praba for the last tip.
Please update the question with the relevant logs and we can take it from there!
