Python profiling, imports (and specially __init__) is what seems to take the most time

Question

I have a script that seemed to run slow and that i profiled using cProfile (and visualisation tool KCacheGrind)

It seems that what is taking almost 90% of the runtime is the import sequence, and especially the running of the _ _ init _ _.py files...

Here a screenshot of the KCacheGrind output (sorry for attaching an image...)

I am not very familiar with how the import sequence works in python ,so maybe i got something confused... I also placed _ _ init _ _.py files in everyone of my custom made packages, not sure if that was what i should have done.

Anyway, if anyone has any hint, greatly appreciated!

---

EDIT: additional picture when function are sorted by self:

---

EDIT2:

here the code attached, for more clarity for the answerers:

from strategy.strategies.gradient_stop_and_target import make_one_trade

from datetime import timedelta, datetime
import pandas as pd
from data.db import get_df, mongo_read_only, save_one, mongo_read_write, save_many
from data.get import get_symbols

from strategy.trades import make_trade, make_mae, get_prices, get_signals, \
    get_prices_subset
#from profilehooks import profile


mongo = mongo_read_only()


dollar_stop = 200
dollar_target = 400
period_change = 3


signal = get_df(mongo.signals.signals, strategy = {'$regex' : '^indicators_group'}).iloc[0]


symbol = get_symbols(mongo, description = signal['symbol'])[0]


prices = get_prices(
    signal['datetime'], 
    signal['datetime'].replace(hour = 23, minute = 59),
    symbol,
    mongo)


make_one_trade(
    signal, 
    prices, 
    symbol,             
    dollar_stop,
    dollar_target,
    period_change)

The function get_prices simply get data from a mongo db database, and make_one_trade does simple calculation with pandas. This never poses problem anywhere else in my project.

---

EDIT3:

Here the Kcache grind screen when i select 'detect cycle' option in View tab:

Could that actually mean that there are indeed circular imports in my self created packages that takes all that time to resolve?

Answer 1

No. You are conflating cumulative time with time spent in the top-level code of the __init__.py file itself. The top-level code calls other methods, and those together take a lot of time.

Look at the self column instead to find where all that time is being spent. Also see What is the difference between tottime and cumtime in a python script profiled with cProfile?, the incl. column is the cumulative time, self is the total time.

I'd just filter out all the <frozen importlib.*> entries; the Python project has already made sure those paths are optimised.

However, your second screenshot does show that in your profiling run, all that your Python code busied itself with was loading bytecode for modules to import (the marshal module provides the Python bytecode serialisation implementation). Either the Python program did nothing but import modules and no other work was done, or it is using some form of dynamic import that is loading a large number of modules or is otherwise ignoring the normal module caches and reloading the same module(s) repeatedly.

You can profile import times using Python 3.7's new -X importtime command-line switch, or you could use a dedicated import-profiler to find out why imports take such a long time.