2

When I write the following code (in ammonite but i don't think it matters)

("tail -f toTail.txt" lineStream) foreach(println(_)), the program give me the last line as intend but then hang, and even if i write more in the file, nothing come out.

How does the API support process that have unbounded output ?

I try to write val myStream = ("tail -f toTail.txt" lineStream_!) but it still does not return write away

Here is what the scala doc says:

lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available.

Hence i don't understand why it blocks

By the way i am having exactly the same behavior with the Ammonite API

If type %%("tail", "-f", "toTail.txt") once again the method just hang and does not return immediately.

pedromss
  • 2,443
  • 18
  • 24
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
  • Describe what you mean by "does not run the right way" – pedromss Sep 03 '17 at 18:54
  • I am expecting an inputStream to be returned. but it does not return it just blocks there. It might be that i am a bit confused as to how the stream class works. The data is supposed to be finite not unbounded for a stream object to be returned? – MaatDeamon Sep 03 '17 at 18:58
  • To my understanding a stream is computer lazily. so it should not be a problem http://docs.scala-lang.org/overviews/collections/concrete-immutable-collection-classes.html#streams – MaatDeamon Sep 03 '17 at 19:01
  • The issue might come from the Scala Process Builder method lineStream – MaatDeamon Sep 03 '17 at 19:01
  • I think, the question is a little unclear (nothing actually "hangs", and nothing is supposed to "return right away" either), but the issue does exist. To clarify: `"tail -f foo.txt".lineStream.foreach(println)` isn't expected to "return right away", it is supposed to sit there, and print out more lines as they are added to `foo.txt`. It does the first thing (sit there), but not the other one - no additional lines are printed out, as they are being added to the file after the command has started. – Dima Sep 04 '17 at 12:26
  • To make it even more clear, you can break it into to commands: `val stream = "tail -f".lineStream` - this returns right away. `stream.foreach(println)` - this prints out the tail of the file, and then blocks, but will not print out more lines as they are added. – Dima Sep 04 '17 at 12:28

2 Answers2

4

There is no issue with the ProcessBuilder (at least not one that stems from your use case). From the ProcessBuilder documentation:

Starting Processes

To execute all external commands associated with a ProcessBuilder, one may use one of four groups of methods. Each of these methods have various overloads and variations to enable further control over the I/O. These methods are:

  • run: the most general method, it returns a scala.sys.process.Process immediately, and the external command executes concurrently.
  • !: blocks until all external commands exit, and returns the exit code of the last one in the chain of execution.
  • !!: blocks until all external commands exit, and returns a String with the output generated.
  • lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available. This method will throw an exception if the return code is different than zero -- if this is not desired, use the lineStream_! method.

The documentation clearly states that lineStream might block until the next line becomes available. Since the nature of tail -f is an infinite stream of lines lineBreak the program will block waiting for the next line to appear.

For the following assume I have a file: /Users/user/tmp/sample.txt and it's contents are:

boom
bar
cat

Why lineStream_! isn't wrong

import scala.language.postfixOps
import scala.sys.process._

object ProcessBuilder extends App {

  val myStream: Stream[String] = ("tail /Users/user/tmp/sample.txt" lineStream_!)
  println("I'm after tail!")
  myStream.filter(_ != null).foreach(println)
  println("Finished")
  System.exit(0)

}

Outputs:

I'm after tail!
boom
bar
cat
Finished

So you see that the lineStream_! returned immediately. Because the nature of the command is finite.

How to return immediately from a command that produces infinite output:

Let's try this with tail -f. You need more control over your process. Again, as the documentation states:

If one desires full control over input and output, then a scala.sys.process.ProcessIO can be used with run.

So for an example:

import java.io.{BufferedReader, InputStreamReader}

import scala.language.postfixOps
import scala.sys.process._

object ProcessBuilder extends App {

  var reader: BufferedReader = _
  try {

    var myStream: Stream[String] = Stream.empty
    val processIO = new ProcessIO(
      (os: java.io.OutputStream) => ??? /* Send things to the process here */,
      (in: java.io.InputStream) => {

        reader = new BufferedReader(new InputStreamReader(in))
        myStream = Stream.continually(reader.readLine()).takeWhile(_ != "ff")
      },
      (in: java.io.InputStream) => ???,
      true
    )
    "tail -f /Users/user/tmp/sample.txt".run(processIO)
    println("I'm after the tail command...")

    Thread.sleep(2000)
    println("Such computation performed while tail was active!")

    Thread.sleep(2000)
    println("Such computation performed while tail was active again!")

    println(
      s"Captured these lines while computing: ${myStream.print(System.lineSeparator())}")

    Thread.sleep(2000)
    println("Another computation!")

  } finally {
    Option(reader).foreach(_.close())
  }
  println("Finished")
  System.exit(0)

}

Outputs:

I'm after the tail command...
Such computation performed while tail was active!
Such computation performed while tail was active again!
boom
bar
cat

It still returns immediately and now it just hangs there waiting for more input. If I do echo 'fff' >> sample.txt from the tmp directory, the program outputs:

Another computation!
Finished

You now have the power to perform any computation you want after issuing a tail -f command and the power to terminate it depending on the condition you pass to the takeWhile method (or other methods that close the input stream).

For more details on ProcessIO check the documentation here.

pedromss
  • 2,443
  • 18
  • 24
  • I think, you are answering a wrong question: the op's issue seems to be that `"tail -f foo.txt".lineStream.foreach(println)` does not print out new lines that are getting added to `foo.txt` after the command has started. This is indeed puzzling ... – Dima Sep 04 '17 at 12:22
  • "Hence i don't understand why it blocks" - I answered why it blocks. And provided alternatives. Maybe he can clarify when he sees this :) – pedromss Sep 04 '17 at 12:31
  • @Dima `"tail -f foo.txt".lineStream.foreach(println)` does print the lines that get sent to the file. :) – pedromss Sep 04 '17 at 12:42
  • Depends on how you send them to the file (and also, probably, on your OS). See my answer ... – Dima Sep 04 '17 at 12:44
  • Maybe. I'm sending them with `echo` and the `>>` operator. Running `El capitan 10.11.6` – pedromss Sep 04 '17 at 12:48
  • yes, that works. but not if you just open the file in editor, type something at the end, and save. That works with `-F`, but not with `-f` – Dima Sep 04 '17 at 13:40
  • @pedromss I commend you for that great answer. Thank you. I do have 1 clarification i would rather further need. "Since the nature of tail -f is an infinite stream of lines lineBreak, the program will block waiting for the next line to appear." It does not says why ? What is the difference between the implementation of LineStream and yours, I don't see why yours returns, and linestream does not. Put another way, what do you not do in your code, that allows you not to block ? – MaatDeamon Sep 06 '17 at 16:27
  • Mine doesn't block to infinite because there is a stop condition on the stream that reads the output from the `tail` command. – pedromss Sep 06 '17 at 16:33
  • @pedromss say i am executing a command that return 6 millions of lines. This is finite not hanging at all. What happen if i use line stream. Is everything loaded in memory ? In the ideal i would like to consume that 6 millions of line in a streaming fashion, not load it in the memory of my program, but continuously get some from the stream, processing it and then move onto the next ones. Could linestream handle that ? Or shall I use process IO for the matter? Beside if i don't know the ending character what are my options? – MaatDeamon Sep 06 '17 at 16:34
  • I could verify that indeed linestream expect the full result to be present before returning a stream over it. I ran into out of memory when executing command that return huge quantity of data. – MaatDeamon Sep 07 '17 at 14:35
  • I guess what I am interested into is the actual inputstream and outputstream of the underlying java sys command execution – MaatDeamon Sep 07 '17 at 14:39
  • @MaatDeamon the `in` parameter of the callback used is the inputstream of the underlying command execution. – pedromss Sep 07 '17 at 16:54
  • I guess if one simply want to chain the result of one command into an Akka-Stream Stream graph, using the in parameter as you pointed out is fine. However if many of those call needs to be done that is when the problems comes, meaning if your stream start with a sys call, then a stage that do some operation and then a sys call within a flatmapContact and so on, in that case, the ProcessIO model does not work. One would have to revert to the java class, because what is needed is that the java.io.stream is returned. But that is the opposite of this model .... Do i get it right here ? – MaatDeamon Sep 08 '17 at 14:43
1

I think, this has to do with how you are adding data to the file, not with the ProcessBuilder. If you are using it in an editor for example to add data, it rewrites the entire file every time you save, with a different inode, and tail isn't detecting that.

Try doing tail -F instead of tail -f, that should work.

Dima
  • 39,570
  • 6
  • 44
  • 70