38

What open-source implementation(s) in C for a pthreads thread pool would you recommend ?

Additional points if this implementation is :

  • Light-weight: glib, APR, NSPR and others come with a big buy-in, I'd rather have just 2 files (header and implementation).
  • Tested on several platforms (Linux, BSD, Mac OS X, etc.).
  • Still maintained.
Mathias Brossard
  • 3,668
  • 2
  • 26
  • 30
  • 9
    I searched forever for a good one to use but ended up having to roll my own. Someone really needs to create a database of low level open source structures written in C. It would save me valuable time and energy. – Swift Jun 09 '11 at 18:42
  • 2
    I've done it too, but I'm hoping to find a better alternative (more platforms, more features, better tested). Good idea for the database, it would save time to a lot of people. – Mathias Brossard Jun 09 '11 at 18:48
  • If/When you find a good implementation, post a link somewhere here so I can check it out. – Swift Jun 09 '11 at 18:50
  • 2
    @Mike Swift: I think Rusty's [CCAN](http://ccan.ozlabs.org/) might fit the bill. – caf Jun 09 '11 at 23:00
  • Maybe the answers in a similar SO question will be of help: http://stackoverflow.com/questions/3853828/a-threadpool-library-in-c – Alexey Kukanov Jun 18 '11 at 20:35
  • @Alexey Kukanov: the link you provide only reference C++ solutions, I'm looking for C ones. – Mathias Brossard Jun 20 '11 at 19:28
  • http://swtch.com/libtask/ - it's not really pthreads, so not putting as an answer, just as an FYI. – Andrew Y Jun 23 '11 at 02:23

4 Answers4

25

I worked on making something I'd be able to use and I've published it on github: it's unimaginably called threadpool.

Mathias Brossard
  • 3,668
  • 2
  • 26
  • 30
  • I've found other pieces of codes that are often duplicated in many project. So I added them in an additional file. It's not compiled yet and I'd like to have a testsuite. This might be a little off-topic, so I'm considering breaking that off to a separate repository. There are no dependencies between the two though. – Mathias Brossard Jun 28 '11 at 15:30
  • Just dump it elsewhere, it's affecting the usability of the project. – Matt Joiner Jun 28 '11 at 23:01
  • Good job. Can't find where to comment in GitHub. – Matt Joiner Jul 01 '11 at 00:31
11

If your goal is light-weight, then the last thing you want is a prewritten, super-general-purpose, high-level-abstraction-based implementation. Implementing a thread pool yourself, suited to your particular task, is fairly trivial, but you might also question whether you actually need a thread pool or whether you'd be fine just creating and destroying threads as needed.

Without knowing more details about your application, I can't give much more specific advice. But the tools you might find useful are:

  • Condition variables
  • Semaphores
  • A job queue protected by a mutex
  • POSIX message queues
R.. GitHub STOP HELPING ICE
  • 208,859
  • 35
  • 376
  • 711
  • 2
    It's a really good point. I've done it already, but I'm hoping that there is a good implementation somewhere that will (a least partially) stop the proliferation of half-baked solutions. – Mathias Brossard Jun 09 '11 at 18:53
  • 1
    @Mathias the half-baked solutions exist because the paradigm itself is half-baked. You will want to look at something like [0MQ](http://www.zeromq.org/) which allows expanding pools across hosts and processes not just threads. – Steve-o Jun 09 '11 at 18:59
  • 2
    I disagree. Working with parallel threads is one of the hardest things. So using a threadpool will save you from huge headaches. – Pithikos Feb 03 '15 at 17:10
  • The advantage of a thread pool (however simple of complex) is you're able to have the job peel off without having to wait to join. Adding a job on the stack is quite helpful and convenient this way. –  Oct 03 '19 at 20:20
  • I think the quality of Apples GCD and Windows Threadpool makes this answer wrong. Also you can't write a good thread pool yourself as it requires the operating system kernel to attach more threads to your system while the system load changes. Using threads is now in this decade hopefully dying. It's just Linux missing out here. With 24 Cores on user desktop system you will otherwise ending up with hunderts of threads in pools (many librareis pulling in their own non standardised). Each one is creating a large non swapable memory fixed kernel stack. – Lothar May 11 '23 at 16:56
5

Here is an implementation with these features:

  • ANSI C and POSIX compliant
  • Minimal but powerful API
  • Synchronisation from the user
  • Full documentation
Pithikos
  • 18,827
  • 15
  • 113
  • 136
2

I once used this, which isn't actually an official implementation per se. It does use pthreads as you requested, and should give you some ideas of what you need to do. (See threadpool.h, threadpool.c, threadpool_test.c, and the Makefile for instructions on how to compile.) You'll obviously have to do some refactoring as it's original intention is probably different than yours. It's commented rather well actually.

Even though this deviates from the original question, I'd also like to mention that the newest C standard, unofficially C1X (see wikipedia, hyperlink limit), has planned support for threads N1570 (google it, hyperlink limit again!) (7.31.15).

Some personal advice from my experience would be to make sure that your application can actually be run in parallel, and if the overhead of creating a new thread is so high that you can't live without a thread pool. Personally I've blundered on both these parts and I've actually ended up with implementations slower than my single threaded application. Also, you might want to be aware of different problems, including cache-lockouts and misses, which would actually degrade the performance of your application.

I'm probably blabbering on by now, but best of luck.

Arka
  • 837
  • 4
  • 8
  • The implementation you point hasn't been update since 2001, I'd say it fails the "still maintained" criteria. It's quite close to the one I refactored mine from (they might share some ancestry). One of the biggest difference is mine uses a pre-allocated FIFO compared to linked list requiring a call to malloc for each entry. The applications I have in mind for using a thread pool scale on multiple processes: currently they create a thread for each task. What I'd like is to have an upper bound to the number of threads running due to external constraints. – Mathias Brossard Jun 24 '11 at 01:36
  • 2
    @Mathias Eh, C code is C code, no matter how old it is. (In fact, I'd consider 2001 pretty new when talking about C.) Anyway, I see you've made your own, which is probably the best choice anyways. – Arka Jun 24 '11 at 01:55