"No module named" error in Apache Storm Python Bolt

Question

I am getting the following error message when I try to locally run a Storm topology with a single bolt implemented in python. I am doing everything like in the example of WordCountTopology in storm-starter kit but it fails to load the modules like matplotlib that are required in my python bolt. Any help or guidance would be appreciated.

I am using Anaconda on Windows machine, if that helps.

12970 [Thread-21-divide] ERROR backtype.storm.daemon.executor - 
java.lang.RuntimeException: Error when launching multilang subprocess
Traceback (most recent call last):
  File "pythonBolt.py", line 4, in <module>
    from matplotlib import mlab
ImportError: No module named matplotlib

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:66) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.daemon.executor$fn__8077$fn__8090.invoke(executor.clj:746) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.util$async_loop$fn__543.invoke(util.clj:473) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
Caused by: java.io.IOException: Die Pipe wurde beendet
    at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_31]
    at java.io.FileOutputStream.write(Unknown Source) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(Unknown Source) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(Unknown Source) ~[na:1.8.0_31]
    at backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
    ... 5 common frames omitted

score 1 · Answer 1 · answered Apr 10 '15 at 08:40

1

Are you using python virtual enviorments? In any case you need to install matplotlib, you can do that with running pip install matplotlib

answered Apr 10 '15 at 08:40

lapinkoira

8,320
9
51
94

I am not using any virtual environment. How would it help? – Erol Apr 10 '15 at 09:06
Well, using virtual enviorments helps to have different python installations in your local machine. For example, you could have the virtual enviorment A where you have a matplotlib version and the virtual enviorment B where you would have a different one. http://docs.python-guide.org/en/latest/dev/virtualenvs/ Anyway, if you just need to install matplotlib locally just run the command I suggested. Maybe you need to install pip first, check this https://pip.pypa.io/en/latest/installing.html – lapinkoira Apr 10 '15 at 09:27
Thanks for the comment. I already have matplotlib installed. The problem I see is that storm is using a standard python interpreter whereas I would like to use the Anaconda version I have on my local machine. I do not know how to change the path or have it pick Anaconda. – Erol Apr 10 '15 at 12:27
Have you tried adding the shebang line on the top of the script? – red_devil Nov 16 '16 at 21:55
1

```#!/home/username/anaconda/bin/python``` – red_devil Nov 16 '16 at 21:55

score 1 · Answer 2 · answered Jul 19 '18 at 07:45

its 2018 and i was stuck for hours for same issue , you need to ensure two things 1) you need to install matplot lib in any of python setups on your machine. so this setup can be done either in default /usr/bin/python or if you have anaconda installation you may do it in /home/user/anaconda3/bin/python or if you have a virtualenv setup then you might install it there. 2)When writing a shell bolt implementation in java code make sure to provide path to python installation where you have installed matplotlib. for eg.below would be code if you installed library in anaconda at /home/user/anaconda3/bin/python

public static class SplitSentence extends ShellBolt implements IRichBolt {

    public SplitSentence() {
      super("home/user/anaconda3/bin/python", "splitsentence.py");
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word"));
    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
      return null;
    }
  }

credits and sources :

In storm, how to specify specific version of python

http://storm.apache.org/releases/2.0.0-SNAPSHOT/Multilang-protocol.html

score 0 · Answer 3 · answered Nov 21 '16 at 14:24

0

Well, I was hit with same issue . I got around by removing anaconda from my supervisor nodes and installing the required packages (i.e - matplotlib,numpy,pandas etc) manually. It's certainly not the best way of doing things, but it works.

answered Nov 21 '16 at 14:24

red_devil

1,009
2
13
23

score 0 · Answer 4 · answered Dec 12 '16 at 05:18

0

Storm use the system default python instead of anaconda python. You installed matplotlib in anaconda python. You did not install matplotlib in system python, so when storm use system python it will show "no module named" error.

answered Dec 12 '16 at 05:18

Shaobo

1
1

"No module named" error in Apache Storm Python Bolt

4 Answers4