0

I have device on bus PCI 09 behind bus 08:01 like this:

0000:08:01.0 -> ../../../devices/pci0000:00/0000:00:03.0/0000:07:00.0/0000:08:01.0/0000:09:00.0

My device in bus 09 off and after reboot CPU, I turn on device on bus 09. So, I rescan PCI bus 0000:08:01.0. But it doesn't find my device.

bus 08:01 after reboot:

root@localhost:~# lspci -s 0000\:08\:01.0 -v
08:01.0 PCI bridge: PLX Technology, Inc. PEX 8614 12-lane, 12-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=08, secondary=09, subordinate=09, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8614 12-lane, 12-Port PCI Express Gen 2 (5.0 GT/s) Switch
Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport

I noticed there isn't "Prefetchable memory behind bridge" field like this:

#lspci -s 0000\:01\:00.0 -v
01:00.0 Class 0604: Device 10b5:8617 (rev ba)
Flags: bus master, fast devsel, latency 0, IRQ 9
Memory at c4b00000 (32-bit, non-prefetchable) [size=128K]
Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
I/O behind bridge: None
Memory behind bridge: c0000000-c4afffff [size=75M]
Prefetchable memory behind bridge: 00000000c4c00000-00000000c96fffff [size=75M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: Device 10b5:8617
Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [148] Virtual Channel
Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport

CPU only find device in bus 09 when I turn on device before reboot. Is there any way for the CPU to find the device on PCI bus after reboot? Or how can I assign memory behind bridge on bus 0000:08:01.0 without device when CPU boots?

banh
  • 11
  • 1
  • 1
    This is not really a programming question. Nevertheless, this question and answer on superuser.com may be relevant: [How can I reserve hotplug bridge's memory only for prefetchable memory using the Linux kernel command line?](https://superuser.com/questions/1054657/). – Ian Abbott Jul 14 '20 at 10:25
  • As of 2018, adding new device numbers was not supported, per https://lwn.net/Articles/767885/ – stark Jul 14 '20 at 13:38
  • How does the system know that you turned on a device? – 0andriy Jul 14 '20 at 20:10
  • Hi @IanAbbott Thank for your comment, my CPU using kernel 4.14,. It seems that is the way I need. I'll try it. – banh Jul 15 '20 at 02:19
  • Hi @stark I reply your comment below. – banh Jul 15 '20 at 02:22
  • Hi @0andriy The system doesn't know, so I tried rescan PCI after boot. According to my new knowledge of pci, CPU won't find device on bus if it is not enumeration at boot. But I see the exception as my comment below. And I want my system can do like that. – banh Jul 15 '20 at 02:29
  • @banh, Are you sure your device is hotplug compatible? Without properly implemented PCI hotplug you would need a lot of quirks. Also try to reproduce on latest kernel (v5.7 or v5.8-rc5). – 0andriy Jul 15 '20 at 19:51

1 Answers1

-1

thank for your answer, I'm just a newbie linux user. I tried rescan bus PCI after boot but not found the device. But in my project, I see an application on system using kernel 4.14 can rescan device on bus PCI and found it after reboot without on device initially.

PCI bus after CPU boots

# ls -l /sys/bus/pci/devices/
lrwxrwxrwx    1 root     root             0 Jan  1 00:50 0000:00:00.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0
lrwxrwxrwx    1 root     root             0 Jan  1 00:50 0000:01:00.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0/0000:01:00.0
lrwxrwxrwx    1 root     root             0 Jan  1 00:50 0000:02:00.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:00.0
lrwxrwxrwx    1 root     root             0 Jan  1 00:50 0000:02:02.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:02.0
lrwxrwxrwx    1 root     root             0 Jan  1 00:50 0000:02:03.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:03.0

Log when CPU boot

pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc3200000-0xc4afffff]
pci 0000:02:03.0:   bridge window [mem 0xc7e00000-0xc96fffff 64bit pref]

I'm sure in bus 5, there is a device ready to communicate with CPU but CPU doesn't find it when boots. But when I rescan bus 02:03 and use bcm_ll_pcie_init, the device found.

After rescan

lrwxrwxrwx    1 root     root             0 Jan  1 01:13 0000:05:00.0 -> ../../../devices/platform/bcm963xx-pcie.0/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:03.0/0000:05:00.0

Log when I rescan bus 02:03

Maple netlink-based raw driver transport driver
bcm963xx-pcie: Core [0] PCIe Intr [PCIE_MSTR_UR_ATTN]
pci 0000:03:00.0: [14e4:6862] type 00 class 0x048000
pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x007fffff 64bit pref]
pci 0000:03:00.0: reg 0x18: [mem 0x00000000-0x00ffffff 64bit pref]
pci 0000:03:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit pref]
pci 0000:03:00.0: PME# supported from D0 D3hot
pcieport 0000:02:00.0: Checking PCIe ASPM for vendor 14e4 device 6862
pcieport 0000:02:00.0: Disabling PCIe ASPM for vendor 14e4 device 6862
pci 0000:04:00.0: [14e4:8470] type 00 class 0x020000
pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x007fffff 64bit pref]
pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
pci 0000:04:00.1: [14e4:8470] type 00 class 0x020000
pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
pci 0000:04:00.1: reg 0x18: [mem 0x00000000-0x007fffff 64bit pref]
pci 0000:04:00.1: PME# supported from D0 D3hot D3cold
pcieport 0000:02:02.0: Checking PCIe ASPM for vendor 14e4 device 8470
pcieport 0000:02:02.0: Disabling PCIe ASPM for vendor 14e4 device 8470
pci 0000:05:00.0: [14e4:6862] type 00 class 0x048000
pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x007fffff 64bit pref]
pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x00ffffff 64bit pref]
pci 0000:05:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit pref]
pci 0000:05:00.0: PME# supported from D0 D3hot
pcieport 0000:02:03.0: Checking PCIe ASPM for vendor 14e4 device 6862
pcieport 0000:02:03.0: Disabling PCIe ASPM for vendor 14e4 device 6862

You can see the device on PCI bus 05, 0000:05:00.0: [14e4:6862] found. I also tried rescan like that in my system but it doesn't work, the device not found. But I noticed my CPU doesn't have "Prefetchable memory behind bridge" like that working system as my post. What do you think about it?

banh
  • 11
  • 1
  • The PCI device can scan the bus and see the new device. the issue is allocating space for it. When I was working on hot-plug for NVMe, we had to make changes to firmware to reserve enough memory and assign it when we plugged NVMe. – stark Jul 15 '20 at 14:32
  • I downvoted the answer because it doesn't answer the question. It's more of an addendum to the question. (But the question also doesn't belong on stackoverflow.) – Ian Abbott Jul 17 '20 at 16:34