2

Using Clojure, I'm pulling some data out of a SQLite DB. It will arrive in the form of a list of maps. Here is an abbreviated sample of what the data looks like.

(
     {:department-id 1 :employee-firstname "Fred" :employee-lastname "Bloggs"}
     {:department-id 1 :employee-firstname "Joe" :employee-lastname "Bloggs"}
     {:department-id 2 :employee-firstname "John" :employee-lastname "Doe"}
      ...
)

I would like to reshape it into something like this:

(
 {:department-id 1 :employees [{:employee-firstname "Joe" :employee-lastname "Bloggs"} {:employee-firstname "Fred" :employee-lastname "Bloggs"}]}
 {:department-id 2 :employees [{:employee-firstname "John" :employee-lastname "Doe"}]
 ... 
)

I know I could a write a function that dealt with the departments and then the employees and "glued" them back together to achieve the shape I want. In fact I did just that in the REPL.

But I've heard a bit about transducers recently and wondered was this an opportunity to use one.

If it is, what would the code look like?

Simon Lomax
  • 8,714
  • 8
  • 42
  • 75
  • 2
    I don't think transducers would really help here, but `group-by` practically does the whole transformation on its own. – noisesmith Feb 12 '15 at 00:17
  • 1
    Thanks for your reply, I know group-by helps here. In my question I hinted that I could write code that could "reshape" the data. In fact I used group-by in one of the steps on the way to reshaping - and I did get the data in the shape I wanted. As a matter of interest though, why wouldn't transducers be a better fit here? I confess I'm struggling to really understand transducers. Oh sure, I can follow tutorials that use simple examples. But when it gets to real world stuff like above - I don't know what to look out for that would indicate - Hey! you should use a transducer here. Any tips? – Simon Lomax Feb 12 '15 at 11:21

1 Answers1

1

I'll have a go, but like many of us, I'm still wrapping my head around this as well.

From my reading on transducers, it would seem the real benefit is in avoiding the need to create intermediate collections, thereby increasing efficiency. This means that to answer your question, you really need to look at what your code will be doing and how it is structure.

For example, if you had something like

(->> (map ....) (filter ..) (map ..) (map ..))

the functions are being run in sequence with new collections being created after each to feed into the next. However, with transducers, you would end up with something like

(->> (map ...) (map ..) (filter ..) (map ...))

where the functions being applied to the data are applied in a pipline fashion on each item from the original collection and you avoid the need to generate the intermediate collections.

In your case, I'm not sure it will help. This is partially because I don't know what other transformations you are applyinig, but mainly because what you are wanting requires a level of state tracking i.e. the grouping of the employee data. This is possible, but I believe it makes it a little harder.

Tim X
  • 4,158
  • 1
  • 20
  • 26
  • Thanks Tim. Again, I'm aware of the theory of transducers and that no intermediate collections are created. Also, you can ignore any "other transformations". As I mentioned above I have written transformations to "reshape" the data, but you can ignore that entirely. I wanted to know if/how this could be achieved using transducers But you hit the nail on the head, the added difficulty here is the state. There are "stateful" transducers which I guess is what's needed. More learning for me to do I think. – Simon Lomax Feb 13 '15 at 09:41
  • OK, then you might find this blog item useful http://clj-me.cgrand.net/2014/10/08/these-arent-the-reducing-functions-you-are-looking-for/ – Tim X Feb 13 '15 at 13:54