2

I'm trying to implement a multi-agent system in C++. To maximize the number of agents per PC, I've thought of the following high-level design:

  • Each agent will be represented as an instance of a class. (In a multi-agent system, an agent is an independent/autonomous entity).

  • Each instantiated agent will be spawned off as a thread.

  • If the OS (Windows or Linux) allows a maximum of A processes, and each process allows spawning of a maximum of B threads, then the total number of agents would be A x B.

  • The agents will communicate with each other using a publisher/subscriber message broker. Plan on using RabbitMQ.

  • For the results to be statistically valid, the requirement is to have a few million agents (e.g. say, 5 million).

Ideally, I'd like each agent to be implemented as one process, instead of one thread. Reasons being, avoiding shared memory bottleneck and interdependency among agents. E.g. A specific thread-crash or a process-crash could cause the entire thread pool to crash with it. An agent represented as a process would not have such limitations.

However, as I understand, the upper limit on the number of processes in an OS is limited (possibly few thousand processes)? If my understanding is correct, the requirement of a few million agents will not work with a process-only strategy.

Question:

What would be the best possible strategy (if at all) to implement such a multi-agent system with multi-million agents on a PC server (say, with an i7 CPU). Essentially, what would be the best implementation strategy that maximizes the number of agents per CPU, without the constraints of shared memory and interdependency?

Thanks in advance.

A.K
  • 91
  • 4
  • 1
    5 million threads? You'll die in the OS scheduler. Use a thread pool, events and reactive. – Remus Rusanu Nov 26 '14 at 17:14
  • 1
    So are agents Objects or Processes? – didierc Nov 26 '14 at 17:17
  • If implemented using threads, they would be objects. I'd like to implement them using processes, though. But that would mean a maximum of only a few of few thousand agents, which (as I understand) is the upper limit of the number of processes that an OS allows? I may be wrong. – A.K Nov 26 '14 at 17:22
  • 1
    What are these agents supposed to be doing? The typical design is to have a collection of agents that you can access by id (a map, perhaps?), and an event dispatcher that reads the queue and calls a processing method on the relevant agent. That way, you can use a thread pool and the number of threads you need is limited to the number of agents currently processing requests. That's the brief description. It's hard to give more concrete suggestions without a better idea of what these agents are doing. – Jim Mischel Nov 26 '14 at 19:29
  • This would be a generic multiagent system, for performing mutiagent based simulation scenarios (e.g. spread of viral flu, or word-of-mouth advertising, experimenting on various optimization algorithms etc.). The agents would be autonomous & independent of each other, and would communicate with each via events (RabbiMQ framework). No agent would have the ability to invoke methods of another agents. Guess a thread pool seems the way to go in this case - though that would mean the limitations of shared-memory & agent dependencies (if one agent crashes, others in the pool would too) would remain. – A.K Nov 27 '14 at 06:12
  • Another way to state this problem is: **How many autonomous entities (implemented as C++ classes; objects or processes), which do not share memory, or are dependent on each other (in the sense, if one crashes, it does not affect others) can be _simultaneously active_ in the memory pool per CPU"** – A.K Nov 27 '14 at 06:22

0 Answers0