4

I tried to follow the drbd Howto "Highly Available NVMe Over Fabrics (NVMe-oF) on RHEL 9", but run into the error

`Error: Unable to find agent 'nvmet-subsystem', try specifying its full name`
`Error: Errors have occurred, therefore pcs is unable to continue`

It happend with the command:
pcs -f nvmet_config resource create p_nvme-subsystem_0 nvmet-subsystem \ nqn=linbit-nqn0 \ op start timeout=10s interval=0 \ op stop timeout=10s interval=0 \ op monitor interval=10s timeout=10s

So I have no experience with pacemaker, but i think it requests the NVMe-oF target export agent, that is located in the package "resource-agents". So I installed it additionally although it was not specified. But no change.
I found this overview according agents:
pcs example for nvme-subsystem
So I changed the command to:
pcs -f nvmet_config resource create nvme-subsystem_0 ocf:heartbeat:nvmet-subsystem nqn=drbd-nvme1 op......
Also not working. I tried this Howto with Rocky 9, Ubuntu Server 22.04 and Debian 11.3. All give the same error message. Clear is that the Howto is not correct, but how can i fix this? I found not one post in the internet according this. Strange. I dont think that i am the first, because Linbit do it already.

kumpel4
  • 41
  • 2

1 Answers1

3

There’s no point to proceed this route either way as DRBD kills NVMe performance during replication. You need some 100% user-mode application doing polling and avoiding interrupt processing overhead, thread context switches, and user-to-kernel (and back) jumps. Nothing living in kernel would fully unleash NVMe potential. Say SPDK NVMe-oF target might be a way better choice.

https://spdk.io/releases/

EDIT: Replaced commercial Lightbits with open-source SPDK.

BaronSamedi1958
  • 13,676
  • 1
  • 21
  • 53
  • no advertising, please – kumpel4 Aug 13 '23 at 20:11
  • 3
    I don’t have anything to do with the Lightbits guys. Replaced the link with an open-source SPDK though. – BaronSamedi1958 Aug 14 '23 at 07:13
  • 1
    Nice to Know there exist a opensource Framework. The performance charts are impressiv. I like that there is support for compression on blocklevel. For my understanding SPDK uses nvme internally so u can not use normal SAS-HDDs. Right? – kumpel4 Aug 14 '23 at 10:50
  • 2
    That was not formulated correctly. I mean SPDK uses the nvme-protocol for "disks". – kumpel4 Aug 14 '23 at 13:57
  • 3
    You can absolutely attach to the aio-exposed Linux kernel block devices. We’ve did it even with a spinning disks (just don’t ask why…), so kernel-managed NVMe shouldn’t be an issue. Just make sure there’s no root file system mounted on top of it. – BaronSamedi1958 Aug 15 '23 at 07:52