2

Prometheus is failing on this error:

Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.098Z caller=main.go:549 msg="Scrape discovery manager stopped"
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.098Z caller=manager.go:888 component="rule manager" msg="Stopping rule manager..."
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.098Z caller=main.go:583 msg="Scrape manager stopped"
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.098Z caller=manager.go:898 component="rule manager" msg="Rule manager stopped"
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.099Z caller=main.go:563 msg="Notify discovery manager stopped"
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.099Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
Aug 04 11:27:36 rockpi prometheus[12548]: level=info ts=2020-08-04T11:27:36.099Z caller=main.go:755 msg="Notifier manager stopped"
Aug 04 11:27:36 rockpi prometheus[12548]: level=error ts=2020-08-04T11:27:36.100Z caller=main.go:764 err="opening storage failed: mmap files, file: /var/lib/prometheus/chunks_head/000022: mmap: invalid argument"
Aug 04 11:27:36 rockpi systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 11:27:36 rockpi systemd[1]: prometheus.service: Failed with result 'exit-code'.

So I found out that below is causing the error but how ?

 --storage.tsdb.path /var/lib/prometheus/

the full command is :

$ prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles

results this error:

panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fe5c48690, 0x14, 0x14, 0x2964a00, 0x400087dc20, 0x2964a00)
    /app/promql/query_logger.go:117 +0x38c
main.main()
    /app/cmd/prometheus/main.go:374 +0x44c8

How do I fix this?

Ciasto piekarz
  • 7,853
  • 18
  • 101
  • 197

3 Answers3

5

I had the same problem and I just emptied chunks_head:

sudo rm -rf /var/lib/prometheus/chunks_head/

The service seemed to start up OK after that and the data looks OK. I'm not sure of the repercussions of this though.

Calum
  • 1,889
  • 2
  • 18
  • 36
0

In my case, Prometheus was ran as a service and systemctrl couldn't start it. I removed the entire directory out of the tree (just in case). This is how I did it:

prometheus[12528]: level=error ts=2021-09-20T06:13:16.727Z caller=main.go:798 err="opening storage failed: mmap files, file: /opt/prometheus/chunks_head/001064: mmap: invalid argument"

That works.

JW Geertsma
  • 857
  • 3
  • 13
  • 19
Yuri Jolobov
  • 31
  • 2
0

In this case, it might be multiple issues or any of these:

  1. Check the user added on prometheus.service file must be correct with proper rights.
  2. Check the rights of the files used by promethues.service

In my case, I have changed the rights of the files and it worked for me.

Sample prometheus.service file:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=<user>
Group=<user_group>
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Provide all access to the required files/folders :

chmod -R 777 /etc/prometheus/
chmod -R 777 /var/lib/prometheus/
Sachin
  • 1,460
  • 17
  • 24
  • recommending chmod 777 without describing the risks , is _very_ bad form. Do you know the risks? – David Thornton Apr 27 '23 at 18:22
  • @DavidThornton as a IT Engineer, we expect the user/developer is already familiar with the linux basics and they know the meaning of these cmds and associated risks. – Sachin Apr 29 '23 at 20:49