3

Good day,

I am currently working on a project where PCIe SSDs are constantly being swapped out and tested through benchmark programs such as VDBench and Iometer. The problem I face right now, which is only on the Linux side (got it working fine on windows), is that if the drives were not on at initial boot-up, they never appear under GParted or Disks. Here's what I have done:

Cold boot, PCIe Add-in-card SSD is off. It is then powered on through a pass through card that is logically controlled to make sure power and shorts are not an issue.

I turn the device on, then run:

sudo sh -c "echo 1 > /sys/bus/pci/rescan"

Performing a

lspci -tv

The device shows with no issues in the tree. When I check under Disks however, it is not there.

I have tried a bunch of different commands with none of them seeming to help me. I have tried

partprobe

Which did not do anything. and:

sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:82:00.0/remove"

Followed up another rescan:

sudo sh -c "echo 1 > /sys/bus/pci/rescan"

As well as:

sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:82:00.0/enable"

Still nothing. Also ran:

dmesg

Which shows, amongst other things:

[   68.128778] pci 0000:82:00.0: [8086:0953] type 00 class 0x010802
[   68.128797] pci 0000:82:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[   68.128820] pci 0000:82:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[   68.133398] pci 0000:84:00.0: [1c58:0003] type 00 class 0x010802
..............................

[   68.141751] nvme 0000:82:00.0: enabling device (0100 -> 0102)
..............................

I do see a lot of fails in dmesg for other addresses, such as:

[ 1264.718446] pcieport 0000:03:04.0: BAR 14: no space for [mem size     0x00400000]
[ 1264.718448] pcieport 0000:03:04.0: BAR 14: failed to assign [mem size 0x00400000]
[ 1264.718451] pcieport 0000:03:04.0: BAR 13: no space for [io  size 0x1000]
[ 1264.718453] pcieport 0000:03:04.0: BAR 13: failed to assign [io  size 0x1000]

Although I have a feeling that those are unrelated to what I am doing, although I'd be happy for someone to prove me wrong.

So, after all of these attempts, does anyone know if there is a way (or if it is even possible) to scan for this PCIe Add-in NVMe SSD and be able to use it without rebooting? I also took a look at some of the threads for other HDDs that reference a rescan for sata based drives, but this is NOT that, so referencing that won't help either.

Thanks in advance.

Trever Wagenhals
  • 381
  • 5
  • 14

3 Answers3

1

I ran into the same issue benchmarking nvme PCIE passthrough with QEMU / Proxmox.

First take note of the driver in use:

lspci -nnk -s '0000:82:00.0'

It should say

Kernel driver in use: vfio-pci

Now unbind the driver, then reprobe:

echo '0000:82:00.0' > /sys/bus/pci/drivers/vfio-pci/unbind
echo '0000:82:00.0' > /sys/bus/pci/drivers_probe

Check the driver again with:

lspci -nnk -s '0000:82:00.0'

Kernel driver in use: nvme

lsblk should now show the drive. Found the procedure here

Andy Fraley
  • 1,043
  • 9
  • 16
0

I tried doing that to save time which is used by rebooting. The PCI device driver at that time was dodgy at best about successfully rescanning and getting all its ducks in a row. The device was an FPGA presenting a proprietary interface device for a device driver I was developing. That was with kernel 2.6.30-something tried around March 2014. My (substandard, but acceptable) solution was to reboot the system.

wallyk
  • 56,922
  • 16
  • 83
  • 148
  • Rebooting isn't a problem now for testing but ultimately the work I'm doing is on a development station that will eventually make it's way into an automated asynchronous test machine where killing power to one of the units that will have multiple devices connected will not be considered asynchronous anymore. Thank you for your feedback though – Trever Wagenhals Mar 15 '16 at 21:26
0

Try adding pci=realloc to your kernel cmdline

S.B
  • 13,077
  • 10
  • 22
  • 49
user3113138
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Apr 13 '23 at 07:40