Restore loki from kubernetes persistent volume

Question

We have deployed loki using helm chart. I have the loki persistent volume. How can I figure out what was the schema config used so that I can retain the logs. Chart loki-2.1.1
Below is the value.yaml

  auth_enabled: false
  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 240h
  schema_config:
    configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v9
      index:
        prefix: index_
        period: 240h

livenessProbe:
  httpGet:
    path: /ready
    port: http-metrics
  initialDelaySeconds: 45
persistence:
  existingClaim: loki-pvc

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "http-metrics"

podManagementPolicy: OrderedReady
readinessProbe:
  httpGet:
    path: /ready
    port: http-metrics
  initialDelaySeconds: 45

Using a pvc in aws. Size=50GB I updated the configuratation. When I run it I see the below error in loki pod.

level=info ts=2022-02-04T16:59:27.774860964Z caller=main.go:128 msg="Starting Loki" version="(version=2.0.0, branch=HEAD, revision=6978ee5d7)"
level=info ts=2022-02-04T16:59:27.774989475Z caller=modules.go:420 msg="RulerStorage is not configured in single binary mode and will not be started."
level=info ts=2022-02-04T16:59:27.77613543Z caller=server.go:225 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=error ts=2022-02-04T16:59:27.776802209Z caller=log.go:149 msg="error running loki" err="mkdir : no such file or directory\nerror creating index client\ngithub.com/cortexproject/cortex/pkg/chunk/storage.NewStore\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/storage/factory.go:176\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:287\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:103\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:204\nmain.main\n\t/src/loki/cmd/loki/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373\nerror initialising module: store\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:105\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:204\nmain.main\n\t/src/loki/cmd/loki/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"

Can anyone help me with the below error.

score 0 · Answer 1 · answered May 24 '23 at 13:58

Not a solution, but this seems to be happening when BoltDB storage is used with local filesystem. (The error message says that directory with name "" cannot be created, see NewBoltDBIndexClient function at pkg/storage/chunk/client/local/boltdb_index_client.go:60 in Loki sources.) Using boldtb-shipper as the storage type instead seem to not produce any error. But I could not figure out how to make the logs available for querying. The queries for the respective time range still return empty data set.

Restore loki from kubernetes persistent volume

1 Answers1