21

For some interaction with a PCI device that is being built, we'd like to create large contiguous pieces of memory that the board can access. As it stands now, the largest piece of memory that I've been able to allocate is 4 megabytes in size. I'm wondering if there are any methods to create larger regions.

I do know that I can use the boot option mem= to do this, but for numa reasons, I'd rather not go this route. If, on the other hand, someone knew a way to do this, but distribute it over numa nodes, that would be fine.

As I said initially, I'm limited to 4 Megabytes currently. Allocations are currently done by __alloc_pages, which is limited by MAX_ORDER. MAX_ORDER is a compile-time constant, and I'm also concerned that editing it could have affects elsewhere.

Thanks.

Bill Lynch
  • 80,138
  • 16
  • 128
  • 173

2 Answers2

9

If you can compile your PCI device driver into the kernel (that is, not linked as a module), you could try allocating the memory at boot time. That should let you bypass the upper bounds on dynamic allocations. Refer to Linux Device Drivers, ed. 3 ch. 8 for details.

Karmastan
  • 5,618
  • 18
  • 24
  • Yeah. I've seen this one before as well. However, I'd really like to be able to do this without recompiling the kernel. It's not something that I'd like to be responsible for. – Bill Lynch May 09 '11 at 20:26
  • @sharth: I think your only other alternative is increasing `MAX_ORDER`. It's your call which kernel recompile is less desirable. – Karmastan May 10 '11 at 02:14
  • As of right now, the plan is to do large amounts of allocation near boot-time. Then, see what parts are contiguous and free back the rest. I'd be a big fan of a more consistant solution though. – Bill Lynch May 10 '11 at 04:40
1

CMA(Contiguous Memory Allocator) are the best solution for your need IMO. You just need to ship to the newest kernel.

Lai Jiangshan
  • 1,420
  • 1
  • 13
  • 23
  • I just searched around for some followup information on this, and I was wondering if you could clarify it. From what I can see, you're required to do a boot-time argument to specify how much of the memory should be reserved. If this is the case, it's similar (but much cleaner, safer and nicer) to the `mem=` option. Also, from what I can see, it's not in 3.4.4 or next-20120713. Has it actually been brought into mainline? – Bill Lynch Jul 14 '12 at 07:13
  • @sharth `mem=` forces the kernel don't access the higher memory and left some memory for the driver's need, but the users need to do more things to manage the left-out memory as far as I known. And that memory can NOT be used even it is free/usable(the same for bootmem). CMA memory can be used by other system(mainly userspace), and if the driver need that memory, the memory management will reclaim the CMA memroy for the driver. Nothing wasted. And CMA is **MERGED** to mainline now, in 3.5, but the newest version is v3.5-rc6, so you may need to wait for 2~3 weeks for 3.5 or use the -rc version – Lai Jiangshan Jul 14 '12 at 09:31
  • You're right. It is in 3.5-rc6. And I absolutely agree that CMA seems to be a much nicer interface than the `mem=` nonsense. And the ability to use unused parts for caching is a neat bonus. Thanks for letting me know about this! – Bill Lynch Jul 15 '12 at 03:52
  • @sharth: let me know(or change your accepted answer) if you finally use CMA. – Lai Jiangshan Jul 17 '12 at 08:08