Densely packed tree of signals

Question

I collect realtime signals, compute derived signals and store both raw and derived data in a circular buffer, so I hold only last million of samples.

Sometimes I need to serialize current values for all signals. So I need something like:

type D0 a = M.Map SignalType D1

data D1 a = D1 
    { foo :: M.Map DoorType a
    , bar :: D2 a
    , baz :: a
    }

data D2 = D2 
    {
        quux :: a
    ,   zoo :: a
    }

data MyData = D0 SignalBuffer 

data CurrentSignals = D0 SignalValue

SignalBuffer is a sequence of SignalValue. It can be an unboxed array of floats. Haskell can derive Functor instances for me, so I can use fmap to fetch last SignalValue from every SignalBuffer and pass the structure to Aeson to serialize.

How do I implement a circular buffer API for SignalBuffer so I can push new values to all the buffers when new ticks arrive? I'd like to conserve memory, so I think I have to use unboxed arrays. Is it advantageous to use mutable unboxed arrays (STUArray?) so array updates don't pile up in memory? Is it possible to use mutable arrays in this setting at all? I'm ready to change MyData and CurrentSignals to whatever does the job.

I know how to implement circular buffers, the question is how to elegantly apply the updates to MyData.

I'm thinking of something like

type UpdateFunc a = MyData -> SignalValue -> Modifier SignalBuffer

updateAllBuffers :: D0 UpdateFunc -> Modifier MyData

Some signals are "convolutions" of other signals (not real convolutions, but a similar kind of processing). To update a buffer for a signal I need to access buffers of other signals - that's why UpdateFunc accepts MyData and SignalValue and returns a buffer modification function.

updateAllBuffers then "zips" D0 UpdateFunc and MyData to get new MyData.

Of course I'm ready to use whatever Modifier fits my task - it can be a function, a monadic value etc.

Do you need to read from just the tail of the buffer, or from possibly any location within it? I wrote about some related work at http://johnlato.blogspot.sg/2012/03/pure-delay-lines.html; if you need random access the best-performing solution I know of is to essentially write it like you would in C, with mutable data in IO. — John L, Jun 08 '12 at 10:39
The question is how to design the API. Haskell is not C, so beautiful generic solutions to bulk updates should be possible even with monadic approach. I cannot use `Seq` because of space overhead, but sequential reading back from the tail should be enough. — nponeccop, Jun 08 '12 at 10:57
I expect you'll end up with something like `type Modifier a = a -> IO a`, but for now I think if you do that `updateAllBuffers` will have to process your maps through a list conversion. That's not necessarily bad, but maybe somebody else will have a good answer. — John L, Jun 08 '12 at 12:57

score 0 · Answer 1 · answered Nov 19 '19 at 20:34

I do not entirely understand what you are trying to do with the code above, but you can use IOVector from Data.Vector.Unboxed.Mutable for a high-performance array to make a circular buffer:

{-# LANGUAGE GADTs #-}

import Data.IORef (IORef, newIORef, readIORef, writeIORef)
import Data.Vector.Unboxed.Mutable (IOVector, Unbox)
import qualified Data.Vector.Unboxed.Mutable as V

data CircularBuffer a where
  CircularBuffer :: Unbox a =>
    { start :: IORef Int -- index for getting an element
    , end   :: IORef Int -- index for putting an element
    , array :: IOVector a
    } -> CircularBuffer a 

newCircularBuffer :: (Unbox a) => Int -> IO (CircularBuffer a)
newCircularBuffer size = CircularBuffer <$> newIORef 0 <*> newIORef 0 <*> V.new size 

putCircularBuffer :: Unbox a => a -> CircularBuffer a -> IO ()
putCircularBuffer newEndValue (CircularBuffer _start end array) = do
  endIndex <- readIORef end
  V.write array endIndex newEndValue
  writeIORef end $! (endIndex + 1) `mod` V.length array

getCircularBuffer :: Unbox a => CircularBuffer a -> IO a
getCircularBuffer (CircularBuffer start _end array) = do
  startIndex <- readIORef start
  startValue <- V.read array startIndex
  writeIORef start $! (startIndex + 1) `mod` V.length array
  pure startValue

You can then make a map like function (it would be IO though) that applies a function to every item in the CircularBuffer's array.

Densely packed tree of signals

1 Answers1