Why this function is not tail-recursive?

Question

I want to read data as it's appearing in a file (the data is written by another process). Something like "continuous reading".

Simple code that just read a file to the end and finishes works without problems:

let readStream (r: StreamReader) =
  seq {
    while not r.EndOfStream do
      yield r.ReadLine() }

Now I add the following logic to it: if it reaches the end of the stream, it waits for a second, then tries again. If the data has not appeared, it sleeps for 2 seconds and so on up to 5 seconds in 5 attempts:

let readStream (r: StreamReader) =
    let rec loop waitTimeout waits =
        seq {
            match waitTimeout with
            | None -> 
                if not r.EndOfStream then 
                    yield r.ReadLine()
                    yield! loop None 0
                else yield! loop (Some 1000) waits
            | Some timeout when waits < 5 -> 
                let waits = waits + 1
                printfn "Sleeping for %d ms (attempt %d)..." timeout waits
                Thread.Sleep (timeout * waits)
                yield! loop None waits
            | _ -> ()
        }
    loop None 0

Unfortunately, it turns out to be not-tail recursive and the app crashes with StackOverflow exception quickly.

Can anybody help with this?

Daniel Fabian · Answer 1 · 2013-11-19T14:03:56.403

1

Before going into further detail, do you have generate tail calls also in debug enabled? (By default F# does not do tail calls in debug code.)

EDIT: Well, your return value is ultimately a seq {...} and not actually the value of loop x y. So you are really building up a nested sequence as opposed to using tail recursion.

Try rewriting your code in such a way, that you do not use a nested sequence, but rather use tail-recursion within the sequence.

edited Nov 19 '13 at 14:03

answered Nov 19 '13 at 12:14

Daniel Fabian

3,828
2
19
28

Switching that flag in the project settings does not change anything. – Vasily Kirichenko Nov 19 '13 at 12:54

score 1 · Answer 2 · answered Nov 19 '13 at 14:52

As Daniel points out, by definition a function with a body that's a sequence expression isn't tail recursive since any recursive calls are delayed. This also guarantees that calling readStream will not cause a stack overflow - it will just return an opaque seq<string> value. So the cause of the stack overflow depends at least partly on the calling code.

score 1 · Answer 3 · answered Nov 20 '13 at 14:29

Hmm, your code as written works fine for me too (on F# 3.0). As kvb points out, is there an issue i your consumer code?

type Reader(k) =
    let mutable n = 0
    member __.EndOfStream = n >= k
    member __.ReadLine() = n <- n + 1; "OK"

let readStream (r: Reader) =
    let rec loop waitTimeout waits =
        seq {
            match waitTimeout with
            | None -> 
                if not r.EndOfStream then 
                    yield r.ReadLine()
                    yield! loop None 0
                else yield! loop (Some 1000) waits
            | Some timeout when waits < 5 -> 
                let waits = waits + 1
                printfn "Sleeping for %d ms (attempt %d)..." timeout waits
                System.Threading.Thread.Sleep(timeout * waits)
                yield! loop None waits
            | _ -> ()
        }
    loop None 0

let test () =
    readStream (Reader 2000000)
    |> Seq.length

Oh. It was my fault. You are absolutely right - the stack overflow was caused by a buggy implementation of Seq.splitEach. I'm so sorry for the misconception. — Vasily Kirichenko, Nov 20 '13 at 14:57

score 0 · Answer 4 · answered Nov 20 '13 at 09:43

I replaced (waitTimeout: TimeSpan option) + pattern matching with (waitTimeout: int) + if...then and the issue has gone. It works OK on seqs of several millions of elements, in constant RAM.

The bad thing is I've no idea why it works.

let readStream (r: StreamReader) =
    let rec loop waitTimeout waits =
        seq {
            if waitTimeout = 0<ms> then
                let line = r.ReadLine()
                if line <> null then
                    yield line
                    yield! loop 0<ms> 0
                else yield! loop 1000<ms> waits
            elif waits < 5 then
                let waits = waits + 1
                Thread.Sleep (int waitTimeout * waits)
                yield! loop 0<ms> waits
        }
    loop 0<ms> 0

Why this function is not tail-recursive?

4 Answers4