0

Trying to profile a python script using cProfile. It has subfiles being called encoder.py and decoder.py in the directory "modules" in "transformer". When I profile it, error shows module error, trasformer is not a module. Though I am not understanding what is the exact issue here.

The header files are as below:

import cProfile
import pstats
import sys, os
from pathlib import Path
sys.path[0] = str(Path(sys.path[0]).parent)


import numpy as np
try:
    import cupy as cp
    is_cupy_available = True
    print('CuPy is available. Using CuPy for all computations.')
except:
    is_cupy_available = False
    print('CuPy is not available. Switching to NumPy.')
    
import pickle as pkl
from tqdm import tqdm
from transformer.modules import Encoder
from transformer.modules import Decoder
from transformer.optimizers import Adam, Nadam, Momentum, RMSProp, SGD, Noam
from transformer.losses import CrossEntropy
from transformer.prepare_data import DataPreparator
import matplotlib.pyplot as plt

when I run the command on terminal:

python -m cProfile -o result.dat transformer.py

this error shows up: ModuleNotFoundError: No module named 'transformer.modules'; 'transformer' is not a package


File "D:\Thesis\numpy-transformer-master\transformer\.\transformer.py", line 28, in <module>
    from transformer.modules import Encoder
ModuleNotFoundError: No module named 'transformer.modules'; 'transformer' is not a package

init.py is already a part of the directory module.

The overall structure of the file is attached as image.

How to resolve this issue The structure of th dirctory in the code is given below

The code structure of sub directories

**The code runs fine and produces desired results if I run without profile command

python transformer.py

Avatar
  • 1
  • 1
  • Do you have a module and a package both named `transformer`? How would python know the difference? – Peter Wood Jun 01 '23 at 11:00
  • I have added a image of the code structure, you could see that the Transformer is a directory and the Modules is another directory in Transformer directory... and a script is transformer.py, that i am trying to profile. – Avatar Jun 01 '23 at 11:44
  • first make sure your virtual environment is activated in the terminal. also you can try changing the `transformer.py` filename to something like `transformers.py`. it makes it a little less confusing to python – se7en Jun 01 '23 at 12:23
  • I have tried it already but still same thingthings! Changed the name to ```transform.py``` as well, but still the same eroor crops up. The Virtual env. is activated already. – Avatar Jun 01 '23 at 13:28
  • the transformer directory is not a package. – Peter Wood Jun 01 '23 at 13:35
  • yes, it is not a package, It is just a directory. but the err. is about ModuleNoFoundError: ```ModuleNotFoundError: No module named 'transformer.modules'; 'transformer' is not a package ``` – Avatar Jun 01 '23 at 13:42

2 Answers2

0

When python sees from transformer.modules import Encoder it looks through every directory specified in sys.path, and for each directory looks for either a MODULE transformer.py or a PACKAGE transformer/__init__.py. You want it to find the PACKAGE, transformer/__init__.py. But, since the error says not a package, I think it is instead finding transformer.py, the MODULE.

So how about if you rename transformer.py to main.py? If you keep getting errors, then it is helpful to print out sys.path right before the import, so you can see where python is about to go look. If you keep having errors, please include that in your followup question.

Or, when you run something with cProfile instead of running it directly with python, then sys.path can be different, which might be the cause of your problem? Since the error says not a package instead of something like could not find, then it appears that your modification of sys.path is working. BUT, to be extra safe, and to be a bit more clear as to what is going on, you could change that sys.path modification to:

# If you move stuff around, then you might have to
# tweak this logic so it actually is the right place
this_dir = Path(__file__).parent
sys.path.append(this_dir)

For more background on how python imports work, read this nice article.

Nick Crews
  • 837
  • 10
  • 13
-1

That show you did not install transformers pachage. Firstly make sure that u installed transformers pachage. If you did not installed , install like: pip install transformers

Myrat_jr
  • 1
  • 3