converting ocaml function to stream

Question

Say I have the following block of OCaml code:

let genblocks () =
  let blocks = ref [] in
  let rec loop depth block =
    let iter i = loop (depth - 1) (i :: block) in
    match depth with
      | 0 -> blocks := block :: !blocks
      | _ -> List.iter iter [1;2;3;4]
  in
  loop 4 [];
  !blocks

This generates a list of lists: [[1;1;1;1]; [1;1;1;2]; ...; [4;4;4;4]].

Now, I want to convert this to a stream (using the Stream module or something similar). However, I'm at a loss as to how to do this while still maintaining the overall recursive structure of the current code. The reason I would like to maintain this structure is that it enables me to easily prune out lists containing certain properties during generation. For example, with this code structure, during generation I can easily prune out, say, lists containing the sublist [1;1]. (The above is just a toy example and my actual application is more complicated...).

Any hints/ideas/pointers on how to convert the above piece of code into a "streaming" instance? I'm sort of stuck with the inability to "backtrack" once depth of zero is hit...

edit: Another way of looking at the problem: is there a way to transform genblocks to get rid of the list reference? This seems like the first step needed to make it stream-compatible.

Thanks.

Answer 1

There are three different things in my answer:

1. a demonstration of a generic technique to get rid of your mutable variable

2. an algorithim-specific technique that makes it very easy to produce a stream

3. a link to a generic technique to turn any producer into an on-demand stream

First, let's make your algorithm generic in the basis of the enumeration:

let genblocks n =
  (* base = [1; ... ; n] *)
  let base = Array.to_list (Array.init n (fun i -> i+1)) in
  let blocks = ref [] in
  let rec loop depth block =
    let iter i = loop (depth - 1) (i :: block) in
    match depth with
      | 0 -> blocks := block :: !blocks
      | _ -> List.iter iter base
  in
  loop n [];
  !blocks

Without looking at what the code does for now, there is a very easy way to get rid of the enumeration: turn any function of type A -> B that uses a mutable type with type C into a function of type A * C -> B * C that receives the state, and returns its modified value -- this is what is called the "state monad". So I will simply add an additional parameter blocks to your functions loop and iter, and make it return not unit but int list list:

let genblocks n =
  let base = Array.to_list (Array.init n (fun i -> i+1)) in
  let rec loop depth blocks block =
    let iter blocks i = loop (depth - 1) blocks (i :: block) in
    match depth with
      | 0 -> block :: blocks
      | _ -> List.fold_left iter blocks base
  in
  loop n [] []

Now let's look at what this algorithm exactly does:

# genblocks 3;;
- : int list list =
[[3; 3; 3]; [2; 3; 3]; [1; 3; 3]; [3; 2; 3]; [2; 2; 3]; [1; 2; 3]; [3; 1; 3];
 [2; 1; 3]; [1; 1; 3]; [3; 3; 2]; [2; 3; 2]; [1; 3; 2]; [3; 2; 2]; [2; 2; 2];
 [1; 2; 2]; [3; 1; 2]; [2; 1; 2]; [1; 1; 2]; [3; 3; 1]; [2; 3; 1]; [1; 3; 1];
 [3; 2; 1]; [2; 2; 1]; [1; 2; 1]; [3; 1; 1]; [2; 1; 1]; [1; 1; 1]]

When called with argument 3 (in your code 4 is hardcoded), this algorithms returns all the 3-combinations of the numbers 1, 2 and 3. Said otherwise, it enumerates all the three-digits numbers in a numeration system in base 3 (using digits between 1 and 3 instead of 0 and 2 as usual).

There is a very simple way to enumerate numbers that you learned in school: to go from a number to the next, simply increment (or decrement) it. In your case, the list starts with the "big" number and goes to the "small" one, so we're going to decrement. With the fact that your base is [1; N] rather than [0; N-1], the decrementation function is written

let decr n block =
  let rec decr n = function
    | [] -> raise Exit
    | 1::rest -> n :: decr n rest
    | i::rest -> (i - 1) :: rest
  in try Some (decr n block) with Exit -> None

I made it return None when we reach 0 (in your system, [1;1;1..]) to easily stop the enumeration at this point.

decr 3 [3;3;3];;
- : int list option = Some [2; 3; 3]
# decr 3 [1;2;3];;
- : int list option = Some [3; 1; 3]
# decr 3 [1;1;1];;
- : int list option = None

From this function it is trivial to enumerate all digits:

let start n = Array.to_list (Array.make n n)

let genblocks n =
  let rec gen = function
    | None -> []
    | Some curr -> curr :: gen (decr n curr)
  in gen (Some (start n))

But the important point is that the whole state of the generation is stored in one value only, the current number. So you can easily turn it into a Stream:

let genblocks n =
  let curr = ref (Some (start n)) in
  Stream.from (fun _ ->
    match !curr with
      | None -> None
      | Some block ->
        curr := (decr n block);
        Some block
  )

# Stream.npeek 100 (genblocks 3);;
- : int list list =
[[3; 3; 3]; [2; 3; 3]; [1; 3; 3]; [3; 2; 3]; [2; 2; 3]; [1; 2; 3]; [3; 1; 3];
 [2; 1; 3]; [1; 1; 3]; [3; 3; 2]; [2; 3; 2]; [1; 3; 2]; [3; 2; 2]; [2; 2; 2];
 [1; 2; 2]; [3; 1; 2]; [2; 1; 2]; [1; 1; 2]; [3; 3; 1]; [2; 3; 1]; [1; 3; 1];
 [3; 2; 1]; [2; 2; 1]; [1; 2; 1]; [3; 1; 1]; [2; 1; 1]; [1; 1; 1]]

Is there a generic way to transform producer-driven functions (that accumulate items at the rythm most natural according to the problem) into consumer-driven function (that produce elements one at the time, when decided by the consumer)? Yes, and I explain it in the following blog post:

Generators, iterators, control and continuations

The big idea is that you can, by mechanic but complex transformations of your code, make explicit what the "context" of the producer is, what his current state, as encoded as a complex mess of values and control flow (which calls have been made, in which conditional branch are you at this point). This context is then turned into a value, that you can use, as the "numbers" used here, to derive a consumer-driven stream.

Answer 2

Assuming that this code isn't homework then you can pretty much translate your code to return a "stream" verbatim by using the Enum module in batteries. Namely the functions Enum.push and Enum.empty. The stream module that comes with OCaml is extremely minimal as you've probably noticed. It's better to use an alternative.

Answer 3

I had a similar problem to yours, so I developed an iterator module (it's definitely not new but I was not able able to find a simple package that did the same thing)

Here is a link : https://github.com/JoanThibault/DAGaml/blob/master/tools/iter.mli

With this module you can re-write your code in

open IterExtra (* add some handy symbols to manipulate iterators *)
let genblock n = (Iter.range 1 (4+1)) $^ n
val genblock : int -> int list Iter.iter

now if you want to filter it with some function prune : int list -> bool :

let genblock n = (Iter.range 1 (4+1)) $^ n |> Iter.filter prune
val genblock : int -> int list Iter.iter

Finally, you can iterate over your iterator with some printing function print : int list -> unit :

let n = 10;;
Iter.iter print (genblock n);;