Can't get Prometheus to install on Kubernetes for Gitlab

Question

bare in mind that I'm new to Kubernetes.

I'm trying to integrate our existing K8 cluster in GitLab. I have added the cluster to gitlab and I can see projects are fetched. However under Health tab I see that I need to install Prometheus.

After trying to install I get

On the cluster this is the error I get

[user]$ kubectl describe pvc prometheus-prometheus-server -ngitlab-managed-apps
Name:          prometheus-prometheus-server
Namespace:     gitlab-managed-apps
StorageClass:  
Status:        Pending
Volume:        
Labels:        app=prometheus
               chart=prometheus-9.5.2
               component=server
               heritage=Tiller
               release=prometheus
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    prometheus-prometheus-server-78bdf8f5b7-dkctg
Events:
  Type    Reason         Age              From                         Message
  ----    ------         ----             ----                         -------
  Normal  FailedBinding  6s (x2 over 6s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

I tried both, specifying storage class and adding persistent volume to no avail, the error remains the same. I can't understand why the volume is not claimed.

This is how the added volume looks like

[user]$ kubectl get pv
NAME                           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                   STORAGECLASS   REASON   AGE
prometheus-prometheus-server   2Gi        RWX            Retain           Available                                           manual                  17m

kubectl describe pv prometheus-prometheus-server
Name:            prometheus-prometheus-server
Labels:          app=prometheus
                 chart=prometheus-9.5.2
                 component=server
                 heritage=Tiller
                 release=prometheus
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"app":"prometheus","chart":"prometheus-9.5.2","compone...
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    manual
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        2Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /var/prometheus-server
    HostPathType:  
Events:            <none>

Answer 1

I'm not quite sure where i found the exact PVC definition for the GitLab managed Prometheus but i am sure that the PVC wants to claim a PV with a capacity of at least 8Gi and an Access Mode of RWO. You need to create a Persistence Volume that meets this criteria (currently you provide 2Gi and RWX).

As far as i experienced it a PVC with RWO does not get assigned to an RWX PV.

Answer 2

Deploying a Prometheus into the gitlab-managed-apps (referenced by the thread from the other answer) is no longer necessary.

Not since GitLab 13.11 (April 2021), which includes:

Bring your own Prometheus for the best GitLab - Kubernetes integrated experience

By integrating your cluster services with GitLab you can benefit from various GitLab features, like Environment boards, Prometheus metrics, and application logs. Previously, these features required you to use GitLab Managed Apps which did not suit the workflow and requirements of many of our users.

With this release, you can integrate with Prometheus and Kubernetes through GitLab services and keep their maintenance on your end, following your own company processes and policies. We provide extensive documentation and a recommended workflow on how to install these applications if you are just getting started. You can still hold the deep metrics integrations available in GitLab as you had with GitLab Managed Prometheus.

See Documentation and Issue.