0

I have a systemd service I am writing. the service code is basically this:

systemd-notify READY=1
while : ; do
  systemd-notify WATCHDOG=1
  sleep 5
done

The service config file has the following configured (among other configurations):

[Service]
...
Type=notify
Restart=always
RestartSec=10
NotifyAccess=all
WatchdogSec=20s
TimeoutStartSec=30s
...

The curies thing is, that when I reboot the system, it seems like systemd is not receiving or handling the systemd-notify READY=1 call. As a result, the service is timed out, and systemd restarts it. On the second time it starts everything is ok and the service launches correctly.

here is what the journalctl for the service reports:

$ journalctl -u my_service.service 
-- Logs begin at Fri 2022-12-30 06:13:51 JST, end at Fri 2022-12-30 06:15:11 JST. --
Dec 30 06:14:00 systemd[1]: Starting my_service startup.sh service...
Dec 30 06:14:30 systemd[1]: my_service.service: Start operation timed out. Terminating.
Dec 30 06:14:30 systemd[1]: my_service.service: Failed with result 'timeout'.
Dec 30 06:14:30 systemd[1]: Failed to start my_service startup.sh service.
Dec 30 06:14:40 systemd[1]: my_service.service: Service hold-off time over, scheduling restart.
Dec 30 06:14:40 systemd[1]: my_service.service: Scheduled restart job, restart counter is at 1.
Dec 30 06:14:40 systemd[1]: Stopped my_service startup.sh service.
Dec 30 06:14:40 systemd[1]: Starting my_service startup.sh service...
Dec 30 06:14:40 systemd[1]: Started my_service startup.sh service.

My systend version is: 237

I have tried adding sleep before sending systemd-notify READY=1. This actually solves the issue but I feel it is just a workaround.

I want to know if I am missing something in working with systemd-notify. Maybe some configuration in the service config file needs to be added? perhaps the correct flow should be to check if systemd is ready to receive notifications before sending them? if so, how do I check this?

Any hint will be much appreciated. Thank you.

einzuk
  • 29
  • 4
  • Have a look at the man page for `sd_notify_barrier()`, i suspect that is the mechanic that prevents the problem you're running into. https://www.man7.org/linux/man-pages/man3/sd_notify_barrier.3.html – Raxi Jan 17 '23 at 14:41
  • Thank you. This was just what i was missing! – einzuk Jan 17 '23 at 15:04
  • From the tag: systemd questions should be for *programming questions* using systemd or its libraries. Questions about *configuring the daemon* (including writing unit files) are better directed to Unix & Linux: https://unix.stackexchange.com. Please delete this. – Rob Jan 17 '23 at 21:59
  • I think you've misread his question; it is about programming an application/script that talks to systemd. – Raxi Jan 17 '23 at 22:04

1 Answers1

1

Following the comment to my question by Raxi I now know that systemd does have some synchronization issues that should be handled directly using sd_notify_barrier().

The bash command I used is: systemd-notify BARRIER=1 And my service looks like this now and launches correctly the first time:

systemd-notify READY=1
systemd-notify BARRIER=1
while : ; do
  systemd-notify WATCHDOG=1
  sleep 5
done
einzuk
  • 29
  • 4