Composing a Buffy buffer from the middle of an array and finding out how much it has consumed

Question

I'd like to use Buffy to interpret binary data starting from the middle of an array. I also need to find out how many bytes of the array have been consumed by Buffy.

Let's say I have a dynamic buffer definition like this:

(ns foo.core
  (:refer-clojure :exclude [read])
  (:use [byte-streams])
  (:require [clojurewerkz.buffy.core :refer :all]
            [clojurewerkz.buffy.frames :refer :all]
            [clojurewerkz.buffy.types.protocols :refer :all])
  (:import [io.netty.buffer Unpooled ByteBuf]))

(def dynbuf
  (let [string-encoder (frame-encoder [value]
                                      length (short-type) (count value)
                                      string (string-type (count value)) value)
        string-decoder (frame-decoder [buffer offset]
                                      length (short-type)
                                      string (string-type (read length buffer offset)))]
    (dynamic-buffer (frame-type string-encoder string-decoder second))))

I hoped I could use a Netty ByteBuf to parse a bunch of bytes using dynbuf starting at an offset:

(def buf
  (let [bytes (concat [0 0 0 4] (map #(byte %) "Foobar"))
        offset 2]
    (Unpooled/wrappedBuffer (byte-array bytes) offset (- (count bytes) offset))))

At this point, I can parse buf per dynbuf:

user> (decompose dynbuf buf)
["Foob"]

At this point, I was hoping that reading the short-type and the string-type from buf has moved its readerIndex by 6, but alas, it is not so:

user> (.readerIndex buf)
0

Is this because buffy/decompose makes some kind of shallow copy of the stream for its internal use, so the readerIndex of the outer buf is not updated? Or am I misunderstanding what readerIndex is supposed to be?

How can I achieve my original goal of passing a (byte-array) at a given offset to Buffy and learning how many bytes it has consumed?

DanLebrero · Accepted Answer · 2016-01-08T09:55:40.810

Buffy is using the absolute version of the getXXX method, which do not modify the position of the buffer, so you cannot use .readerIndex.

I see two possible options, depending on what you are trying to achieve:

Use Buffy dynamic frames. Note that the clojurewerkz.buffy.frames namespace has a decoding-size function if you want to know how much the dynamic frame will take. Something like:

(defn read-from-middle [data f-type start-idx]
  (let [tmp-buf (Unpooled/wrappedBuffer data start-idx (- (alength data) start-idx))
        buffy-buffer (dynamic-buffer f-type)
        total-size (decoding-size f-type tmp-buf 0)]
    [(decompose buffy-buffer tmp-buf) (+ start-idx total-size)]))

(def f-type
  (let [string-encoder (frame-encoder [value]
                                      length (short-type) (count value)
                                      string (string-type (count value)) value)
        string-decoder (frame-decoder [buffer offset]
                                      length (short-type)
                                      string (string-type (read length buffer offset)))]
    (frame-type string-encoder string-decoder second)))

(let [my-data (byte-array [0 1 0x61 0 2 0x62 0x63 0 1 0x64])
      idx 0
      [i1 idx] (read-from-middle my-data f-type idx)
      [i2 idx] (read-from-middle my-data f-type idx)
      [i3 idx] (read-from-middle my-data f-type idx)]
  [i1 i2 i3])

Calculate the size of the frame as Buffy is doing and manually set the correct position in the buffer. Something like:

(import [io.netty.buffer Unpooled])
(require '[clojurewerkz.buffy.core :as buffy]
         '[clojurewerkz.buffy.types.protocols :as ptypes])

(defn read-from-middle [data spec start-idx]
  (let [total-size (reduce + (map ptypes/size (map second spec)))
        tmp-buf (Unpooled/wrappedBuffer data start-idx (- (alength data) start-idx))
        buffy-buffer (buffy/compose-buffer spec :orig-buffer tmp-buf)]
    [(buffy/decompose buffy-buffer) (+ start-idx total-size)]))

(let [my-data (byte-array [0 0 0 1 0 0 0 2 0 0 0 3])
      spec (buffy/spec :foo (buffy/int32-type))
      idx 0
      [i1 idx] (read-from-middle my-data spec idx)
      [i2 idx] (read-from-middle my-data spec idx)
      [i3 idx] (read-from-middle my-data spec idx)]
   [i1 i2 i3])

Composing a Buffy buffer from the middle of an array and finding out how much it has consumed

1 Answers1