process is not spawning with Multiprocessing module in celery

Question

I have a pretty standard set-up: Django + Rabbitmq + Celery.

I am trying to spawn a process using standard python multiprocessing module in celery.

I have noticed that process itself is not spawning. What could be the reason for not spawning the process. here is the code:

import multiprocessing as mp
from celery.schedules import crontab
from celery.decorators import periodic_task

@periodic_task(run_every=crontab(minute='*/1'), name='test_process_celery')
def main():
data = config_read()
try:
    myqueue = mp.Queue(-1)
    mylog_process = mp.Process(target=test_logger_process, args=(myqueue,))
    mylog_process.start()
    . . .
    . . .
except Exception as e:
    raise
finally:
    mylog_process.join()

1. Can we use multiprocessing module in celery?

2. Is there any celery limitation with multiprocessing ? if there is no limitation why process is not spawned?

Thanks you in Advance.

Answer 1

try this:

@periodic_task(run_every=crontab(minute='*/1'), name='test_process_celery')
def main():
    from multiprocessing import current_process
    current_process().daemon = False

    your_stuff()

    current_process().daemon = True