1

I have a requirement where I need to read text file then transform it and write it to some other file. I wish to do this in parallel fashion like one thread for read, one for transform and another for write.
Now to share data between threads I need some channel, I was thinking to use BlockingQueue for this but would like to explore some other (better) alternatives if available.
Guava has a EventBus but not sure whether this is a good fit for the requirement. What other alternatives are available and which one is best from performance point of view.

Premraj
  • 7,802
  • 8
  • 45
  • 66

1 Answers1

2

Unless your transform step is really intensive, this is probably a waste of time.

Think of it this way. What are you asking for?

You're asking for something that

  • Takes an incoming stream of data
  • Copies it to another thread
  • Presents it to that thread as an incoming stream of data

What data structure best represents an incoming stream of data for step 3? (Hint: it's the InputStream you started with!)

What value do the first two steps add? The "transform" thread can read from disk just as fast as it could read from disk through another thread. Adding the thread inbetween does not speed up the disk read.

You would start to consider adding another thread when

  1. Your problem can be usefully divided into independent pieces of work (say, each thread works on a chunk of text
  2. The cost of splitting the problem into those pieces of work is significantly smaller than the overhead of adding an additional thread and coordinating between them (which is small, but not free!)
  3. The problem requires more resources than a single CPU can provide (a thread gives you access to more CPU resources, but doesn't provide much value in terms of I/O throughput)
Steven Schlansker
  • 37,580
  • 14
  • 81
  • 100
  • 1
    Transform is little intensive (as it includes tons of validations, conversions, filtration etc) and cannot be clubbed with reader as it follows some specific rules to read. – Premraj Feb 08 '12 at 08:18
  • Then you probably need to describe your problem a bit better to get useful responses. I think as stated it is too generic of a problem to suggest concrete improvements. – Steven Schlansker Feb 08 '12 at 18:28