How to build an infinite tree with duplicate elimination via cache of weak pointers in Haskell

Question

The following code builds up an infinite tree, while at the same time creating a cache of all subtrees, such that no duplicate subtrees are created. The rationale for elimination of duplicate subtrees comes from the application to state trees of chess-like games: one can often end up in the same game state by just changing the order of two moves. As the game progresses, states that become inaccessible should not continue to take up memory. I thought I could solve that problem through the use of weak pointers. Unfortunately using weak pointers brings us into the IO Monad and this seems to have destroyed enough/all lazyness such that this code does not terminate any more.

My question is thus: Is it possible to efficiently generate a lazy (game state) tree without duplicate subtrees (and without leaking memory)?

{-# LANGUAGE RecursiveDo #-}

import Prelude hiding (lookup)
import Data.Map.Lazy (Map, empty, lookup, insert)
import Data.List (transpose)

import Control.Monad.State.Lazy (StateT(..))
import System.Mem.Weak
import System.Environment

type TreeCache = Map Integer (Weak NTree)

data Tree a = Tree a [Tree a]
type Node = (Integer, [Integer])
type NTree = Tree Node

getNode (Tree a _) = a
getVals = snd . getNode

makeTree :: Integer -> IO NTree
makeTree n = fst <$> runStateT (makeCachedTree n) empty

makeCachedTree :: Integer -> StateT TreeCache IO NTree
makeCachedTree n = StateT $ \s -> case lookup n s of
  Nothing -> runStateT (makeNewTree n) s -- makeNewTree n s                                                                                                                                   
  Just wt -> deRefWeak wt >>= \mt -> case mt of
    Nothing -> runStateT (makeNewTree n) s
    Just t -> return (t,s)

makeNewTree :: Integer -> StateT TreeCache IO NTree
makeNewTree n = StateT $ \s -> mdo
  wt <- mkWeak n t Nothing
  (ts, s') <- runStateT
              (mapM makeCachedTree $ children n)
              (insert n wt s)
  let t = Tree (n, values n $ map getVals ts) ts
  return (t, s')

children n = let bf = 10 in let hit = 2 in [bf*n..bf*n+bf+hit-1]

values n [] = repeat n
values n nss = n:maximum (transpose nss)

main = do
  args <- getArgs
  let n = read $ head args in
    do t <- makeTree n
       if length args == 1 then putStr $ show $ take (fromInteger n) $ getVals t else putStr "One argument only!!!"

Answer 1

My question is thus: Is it possible to efficiently generate a lazy (game state) tree without duplicate subtrees (and without leaking memory)?

No. Essentially what you're trying to do is use transposition tables to memoize your tree search function (e.g. negascout). The problem is that games have an exponential number of states and maintaining a transposition table which remembers all the transpositions of the state space is infeasible. You just don't have that much memory.

For example, Chess has a state space compexity of 10^47. That's several orders of magnitude greater than all the computer memory available in the entire world. Of course, you can reduce this amount by not storing reflections (Chess has 8 reflection symmetries). Furthermore, many of these transpositions would be unreachable due to pruning of the game tree. However, the state space is so intractable that you would still never to able to store every transposition.

What most programs usually do is they use a transposition table of a fixed size and when two transpositions hash to the same value, they use a replacement scheme to decide which entry will be kept and which one will be discarded. This is a tradeoff because you are keeping the value which you think will be most efficient (i.e. most visited or closer to the root node) at the cost of having to traverse the other transposition. The point is that it's impossible to generate a game tree which has no duplicate subtrees unless you're from a sufficiently advanced alien civilization.