3

Context : Trying to use prometheus on k8s 1.10.2 and using azure file storage as persistent storage medium

Problem : using azurefile storage with prometheus gives me the following error :

level=info ts=2018-06-29T11:08:50.603235475Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2018-06-29T11:08:50.603302775Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2018-06-29T11:08:50.603341576Z caller=main.go:217 host_details="(Linux 4.15.0-1013-azure #13~16.04.2-Ubuntu SMP Wed May 30 01:39:27 UTC 2018 x86_64 prometheus-84f89cd668-r8p5r (none))"
level=info ts=2018-06-29T11:08:50.605677083Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-29T11:08:50.605759983Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2018-06-29T11:08:50.605816483Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=error ts=2018-06-29T11:08:50.778059828Z caller=main.go:323 msg="Opening storage failed" err="open DB in /home/prometheus: Lockfile created, but doesn't exist"

Note : I do not want to use the --storage.tsdb.no-lockfile flag on the prometheus deploymenet

Is there any other way i can fix this issue?

Thanks for any input on how to fix this issue.

moonkotte
  • 3,661
  • 2
  • 10
  • 25
D007
  • 31
  • 1
  • 4
  • If AzureFile is Azure's equivalent of S3, you _for sure_ do not want to run any kind of database on top of an _eventually consistent_ data layer, or it'll do exactly what you just described – mdaniel Jun 30 '18 at 07:29
  • What other storage options from azure would you suggest for this implementation? – D007 Jul 02 '18 at 09:47
  • As best I can tell, [Data Disk](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/about-disks-and-vhds#data-disk) which has a `volume` subkey of of [`azureDisk`](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_disk) but since I'm not in Azure, I can't test it for sure – mdaniel Jul 03 '18 at 02:39
  • Just store data in [remote storage](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage). In this case the Prometheus would become stateless service, since all the local data is automatically replicated in the remote storage. – valyala Oct 29 '18 at 10:28
  • Already trying it on a remote storage.. The problem is that we have to define what type remote storage to be used. And I tried, remote nfs and remote azure storage. although with nfs it worked a bit better, azure storage type just cannot handle lock files. – D007 Oct 30 '18 at 11:04
  • I think the alternative to disabling the locking would be to use a different deployment strategy `.spec.strategy.type==Recreate` which would shutdown all existing servers before creating the new ones. That should cause the DB to get unlocked then the new server will re-lock it. – justin.m.chase Apr 14 '21 at 17:25

0 Answers0