How install Ceph on a kubernetes cluster

Question

we want to use Ceph but we want to use Docker and Kubernetes to deploy new instances of Ceph quickly.

I tried to use the default ceph docker hub: ceph/daemon-base. But I didn't work.

I tried to use the ceph-container. Seems like it doesn't work.

This is my last deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ceph3-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ceph3
  template:
    metadata:
      labels:
        app: ceph3
    spec:
      containers:
        - name: ceph
          image: ceph/daemon-base:v3.0.5-stable-3.0-luminous-centos-7
          resources:
            limits:
              memory: 512Mi
              cpu: "500m"
            requests:
              memory: 256Mi
              cpu: "250m"
          volumeMounts:
            - mountPath: /etc/ceph
              name: etc-ceph
            - mountPath: /var/lib/ceph
              name: lib-ceph
      volumes:
        - name: etc-ceph
          hostPath:
            path: /etc/ceph
        - name: lib-ceph
          hostPath:
            path: /var/lib/ceph

Does someone already install a ceph instance on Kubernetes?

I tried to follow the tutorial here

But pods not working:

pod/ceph-mds-7b49574f48-vhvtl         0/1     Pending            0          81s
pod/ceph-mon-75c49c4fd5-2cq2r         0/1     CrashLoopBackOff   3          81s
pod/ceph-mon-75c49c4fd5-6nprj         0/1     Pending            0          81s
pod/ceph-mon-75c49c4fd5-7vrp8         0/1     Pending            0          81s
pod/ceph-mon-check-5df985478b-d87rs   1/1     Running            0          81s

Answer 1

The common practice for deploying stateful systems on Kubernetes is to use an Operator to manage and codify the lifecycle management of the application. Rook is an operator that provides Ceph lifecycle management on Kubernetes clusters.

Documentation for using Rook to deploy Ceph clusters can be found at https://rook.io/docs/rook/v1.1/ceph-storage.html

For a basic introduction, you can use the Rook Storage Quickstart guide

The core Ceph team is highly involved in working on Rook and with the Rook community, and Rook is widely deployed within the Kubernetes community for distributed storage applications, and the Ceph Days event now has added Rook] explicitly to become Ceph + Rook Days

Answer 2

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:ceph:default" cannot list resource "pods" in API group "" in the namespace "ceph".

This error means Your default user doesn't have an access to resources in default namespace, look into documentation how to deal with rbac and how to grant user accesses. There is the same error on github.

Install Ceph

To install ceph I would recommend You to use helm

Install helm

There are few ways to install helm, You can find them there.

Personally i am using helm github releases to download latest version. When answering this question latest release was 2.15.0. That's how You can install it.

1.Download helm (If you want to install current version of helm, check above helm github releases link, there might be a new version of it. Everything you need to do then is change v2.15.0 in wget and tar command to new, for example v2.16.0)

wget https://get.helm.sh/helm-v2.15.0-linux-amd64.tar.gz

2.Unpack it

tar -zxvf helm-v2.15.0-linux-amd64.tar.gz

3.Find the helm binary in the unpacked directory, and move it to its desired destination

mv linux-amd64/helm /usr/local/bin/helm

4.Install tiller

The easiest way to install tiller into the cluster is simply to run helm init. This will validate that helm’s local environment is set up correctly (and set it up if necessary). Then it will connect to whatever cluster kubectl connects to by default (kubectl config view). Once it connects, it will install tiller into the kube-system namespace.

And then follow official documentation to install Ceph.