5

I changed the class WordCount in WordCountTopology as follows:

public static class WordCount extends BaseBasicBolt {
    Map<String, Integer> counts = new HashMap<String, Integer>();

    @Override
    public void execute(Tuple tuple, BasicOutputCollector collector) {
        String word = tuple.getString(0);
        Integer count = counts.get(word);
        if(count==null) count = 0;
        count++;
        counts.put(word, count);
        OutputStream o;
        try {
            o = new FileOutputStream("~/abc.txt", true);
            o.write(word.getBytes());
            o.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        collector.emit(new Values(word, count));
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("word", "count"));
    }
}

in which I write the word to file abc.txt.

When I ran the WordCountTopology in local mode(which used the LocalCluster), it just works fine. But when running in distributed mode(which used the StormSubmitter.submitTopology() method), the WordCount class didn't write the word to abc.txt as if the execute() method has not run at all. Could anyone give me some idea? Thanks a lot!

P.S. I'm sure my nimbus, supervisor, ui, zookeeper is running normally, and I can see the task in 127.0.0.1:8080.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Judking
  • 6,111
  • 11
  • 55
  • 84
  • 2
    do you have permissions to write to this directory? What's your UI shows, does your bolt receives tuples? – Vor Jul 24 '13 at 00:39
  • are you sure that the task is not executed on another machine? – ChrisBlom Aug 30 '13 at 22:35
  • put a logger to check what values you receive in `word.getBytes()` call – user2720864 Sep 06 '13 at 08:38
  • You will have to do some basic debugging like adding some logs around your code. As @Vor said, this is most probably a permissions issue since the code is running in another supervisor machine. – Munim Sep 06 '13 at 08:47
  • I tried logging (via org.slf4j.Logger) in my bolts/spouts and that does not show up in the worker logs either. – Adrian Dec 12 '13 at 17:43

1 Answers1

3

The main problem is location of abc.txt file.This file will be created in the system from where you are submitting the topology.So this file wont be available in other cluster machines.You can check supervisors log for file not found error.To resolve this issue you need some NFS configuration through which common location can be shared by all cluster machines.After configuring NFS create new file in common location so that this file wound be available for all supervisors.

user2435082
  • 295
  • 5
  • 16