Why does the validation fail (ExceptionInfo Value does not match schema) for a prismatic schema using onyx?

Question

Can someone throw light on whats going on here ? I have double checked dynamodb-local behavior using a non-onyxian client and simple operations work using Faraday . The code which I am trying to run is here and I have logged an issue here but haven't got any response.

Here is the output from the REPL:

$ lein repl
WARNING: cat already refers to: #'clojure.core/cat in namespace: clj-uuid.bitmop, being replaced by: #'clojure.core.reducers/cat
nREPL server started on port 44195 on host 127.0.0.1 - nrepl://127.0.0.1:44195
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.7.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b18
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

cqrs-server.core=> (start)
ExceptionInfo Value does not match schema: {:onyx/consumption missing-required-key, :onyx/batch-size missing-required-key, :onyx/medium missing-required-key}  schema.core/validate (core.clj:161)

cqrs-server.core=>

The full stacktrace using *e is below:-

cqrs-server.core=> *e
#error {
 :cause "Value does not match schema: {:onyx/consumption missing-required-key, :onyx/batch-size missing-
required-key, :onyx/medium missing-required-key}"
 :data {:type :schema.core/error, :schema (conditional #object[onyx.validation$fn__16409 0x6ba7a7e4 "ony
x.validation$fn__16409@6ba7a7e4"] {:onyx/name Keyword, :onyx/type (enum :output :function :input), :onyx
/consumption (enum :concurrent :sequential), :onyx/batch-size (pred pos?), Keyword Any, :onyx/medium Key
word} #object[clojure.core$constantly$fn__4383 0x5a6f89f1 "clojure.core$constantly$fn__4383@5a6f89f1"] {
:onyx/name Keyword, :onyx/type (enum :output :function :input), :onyx/consumption (enum :concurrent :seq
uential), :onyx/batch-size (pred pos?), Keyword Any, :onyx/fn Keyword}), :value {:onyx/name :event/aggre
gate-out, :onyx/type :output}, :error {:onyx/consumption missing-required-key, :onyx/batch-size missing-
required-key, :onyx/medium missing-required-key}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Value does not match schema: {:onyx/consumption missing-required-key, :onyx/batch-size miss
ing-required-key, :onyx/medium missing-required-key}"
   :data {:type :schema.core/error, :schema (conditional #object[onyx.validation$fn__16409 0x6ba7a7e4 "o
nyx.validation$fn__16409@6ba7a7e4"] {:onyx/name Keyword, :onyx/type (enum :output :function :input), :on
yx/consumption (enum :concurrent :sequential), :onyx/batch-size (pred pos?), Keyword Any, :onyx/medium K
eyword} #object[clojure.core$constantly$fn__4383 0x5a6f89f1 "clojure.core$constantly$fn__4383@5a6f89f1"]
 {:onyx/name Keyword, :onyx/type (enum :output :function :input), :onyx/consumption (enum :concurrent :s
equential), :onyx/batch-size (pred pos?), Keyword Any, :onyx/fn Keyword}), :value {:onyx/name :event/agg
regate-out, :onyx/type :output}, :error {:onyx/consumption missing-required-key, :onyx/batch-size missin
g-required-key, :onyx/medium missing-required-key}}
   :at [schema.core$validate invoke "core.clj" 161]}]
 :trace
 [[schema.core$validate invoke "core.clj" 161]
  [onyx.validation$validate_catalog invoke "validation.clj" 48]
  [onyx.validation$validate_job invoke "validation.clj" 122]
  [onyx.api$submit_job invoke "api.clj" 90]
  [cqrs_server.core$start invoke "core.clj" 207]
  [cqrs_server.core$eval19231 invoke "form-init17061160983479843.clj" 1]
  [clojure.lang.Compiler eval "Compiler.java" 6782]
  [clojure.lang.Compiler eval "Compiler.java" 6745]
  [clojure.core$eval invoke "core.clj" 3081]
  [clojure.main$repl$read_eval_print__7099$fn__7102 invoke "main.clj" 240]
  [clojure.main$repl$read_eval_print__7099 invoke "main.clj" 240]
  [clojure.main$repl$fn__7108 invoke "main.clj" 258]
  [clojure.main$repl doInvoke "main.clj" 258]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__18624 invoke "interruptible_eval.clj"
87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invoke "core.clj" 630]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1868]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 85]
  [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__18669$fn__18672 invoke "inte
rruptible_eval.clj" 222]
  [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__18664 invoke "interruptible_eval.clj"
190]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}
cqrs-server.core=>
cqrs-server.core=>

The output from a Faraday client REPL (my-ns is a local namespace for testing) which will connect to the same dynamodb running locally is below:

my-ns=> (far/list-tables client-opts)
(:events)

my-ns=> (far/describe-table client-opts :events)
{:lsindexes nil, :gsindexes [{:name :event-idx, :size 0, :item-count 0, :key-schema [{:name :type, :type :hash} {:name :date, :type :range}], :projection {:projection-type "ALL", :non-key-attributes nil}, :throughput {:read 1, :write 1, :last-decrease nil, :last-increase nil, :num-decreases-today nil}}], :name :events, :throughput {:read 1, :write 1, :last-decrease #inst "1970-01-01T00:00:00.000-00:00", :last-increase #inst "1970-01-01T00:00:00.000-00:00", :num-decreases-today 0}, :prim-keys {:id {:key-type :hash, :data-type :s}, :date {:key-type :range, :data-type :n}, :type {:data-type :s}}, :size 0, :status :active, :item-count 0, :creation-date #inst "2016-03-10T22:53:33.601-00:00", :indexes nil}
my-ns=>

my-ns=> (far/scan client-opts :events)
[]

I have created a public gist for core.clj here and for cqrs.clj here for review. My project.clj looks like this.

Can someone please help me fix this ?
TIA

This is the only stacktrace I have on the REPL. I can update with lines before and after if that's what you are looking for. I have updated the github issue as well with some more detail if you can click on the issue link above (main question) including output from the log file output for cqrs.log and the onyx.log. — ZeroGraviti, Mar 11 '16 at 13:07
When you call `(start)` in your REPL and you get an exception you can then call `*e` which will print the last thrown exception with its whole stacktrace. — Piotrek Bzdyl, Mar 11 '16 at 14:08
I have updated the main question post with the output of `*e`. — ZeroGraviti, Mar 11 '16 at 14:26
Thanks for the stacktrace. One more request: I checked the repo at https://github.com/Yuppiechef/cqrs-server but there is no `cqrs-server.core` namespace and `start` function (with zero arity). Where is the code you are running? Could you publish it on github? I suspect there might be an issue in that `start` function that it doesn't pass a correct parameters to `onyx.api/submit_job` function as the exception stacktrace suggests. — Piotrek Bzdyl, Mar 11 '16 at 16:26
I have checked out from that repo based off commit id https://github.com/Yuppiechef/cqrs-server/commit/08707367ba0a90bbf8ece95624dd365896a4cc47. I have mentioned this in the Issue #8 [https://github.com/Yuppiechef/cqrs-server/issues/8] created on that repo. All my changes are in my local branch currently. I had already created gists of the files which I think are relevant and updated in the question post above. I can create/share more public gists of files which you might find interesting. Please let me know. — ZeroGraviti, Mar 11 '16 at 17:00
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/106040/discussion-between-zerograviti-and-piotrek-bzdyl). — ZeroGraviti, Mar 11 '16 at 17:06

Piotrek Bzdyl · Accepted Answer · 2016-03-11T20:31:38.113

Based on the info you provided and our chat the answer straight to your question is that the last entry in the :catalog vector from the configuration is missing keys as indicated by the schema validation error:

{:onyx/consumption missing-required-key, :onyx/batch-size missing-required-key, :onyx/medium missing-required-key}

The offending entry (last one in the vector) is:

{:onyx/name :event/aggregate-out, :onyx/type :output}

And you need to fix it so it contains those missing keys:

{:onyx/name :event/aggregate-out,
 :onyx/type :output,
 :onyx/consumption <some value here>,
 :onyx/batch-size <some value here>,
 :onyx/medium <some value here>}

But in general I would rethink your general approach. What goal are you trying to achieve?

In my opinion it is not worth to struggle with getting to work a framework basing on a "randomly" selected old commit which seems to contain code in a broken state. How are you going to support your application if you discover a bug in cqrs-server? Will you fork it and support it yourself starting from that custom commit?

I would rather start with the requirements you have for your problem and then evaluate the solutions using more reliable and better supported library/framework so you have a better chance supporting your project.

This is more of a learning project to understand how to implement cqrs+es. There is a lot of .NET stuff out there and I was trying to understand how to implement this in something closer to java and get to learn a little bit more of clojure :) — ZeroGraviti, Mar 11 '16 at 20:30
If this is a "throw away" project, than it's OK from my stand point. However, starting with an already big and complex solution is very ambitious :) — Piotrek Bzdyl, Mar 11 '16 at 20:33
Fixed and it works now (as per the documented use cases). I will see how I can create a PR or fork this since most of the changes are dependent on a mix-n-match of version of various libraries). Thanks for your guidance !!! — ZeroGraviti, Mar 11 '16 at 21:16

Why does the validation fail (ExceptionInfo Value does not match schema) for a prismatic schema using onyx?

1 Answers1