4

I have my data in a propriety format, None of the ones supported by Apache drill. Are there any tutorial on how to write my own storage plugin to handle such data.

sushil
  • 165
  • 1
  • 9
  • 1
    Is it some kind of RDBMS? If yes, you can define your plugin. If this is something else, you have to write code for that plugin. – Dev Jun 01 '16 at 05:04
  • Even though its not an RDBMS we do have tools that support SQL like language on top of that data set. – sushil Jun 01 '16 at 17:54

1 Answers1

12

This is something that really should be in the docs but currently is not. The interface isn't too complicated, but it can be a bit much to look at one of the existing plugins and understand everything that is going on.

There are 2 major components to writing a storage plugin, exposing information to the query planner and schema management system and then actually implementing the translation from the datasource API to the drill record representation.

The Kudu plugin was added recently and is a reasonable model for a storage system with a lot of the elements Drill can take advantage of. One thing I would note is that if your storage system is not distributed and you just plan on making all remote reads you don't have to do as much work around affinities/work lists/assignments in the group scan. If I have some time soon I'll try to write up a doc on the different parts of the interface and maybe write a tutorial about one of the existing plugins.

https://github.com/apache/drill/tree/master/contrib/storage-kudu/src/main/java/org/apache/drill/exec/store/kudu

jaltekruse
  • 313
  • 2
  • 5
  • This is the exact information I was looking for. I did go through the Hiva storage plugin and found it too complicated. I have a followup question. How much of a work is to actually write a plugin/module to do the actual query execution on the end worker nodes. I mean instead of just reading and providing the record, i want to take drills execution plan and execute it. – sushil Jun 06 '16 at 21:46
  • @jaltekruse, I want to build as well a storage plugin and would appreciate any tutorial about this. Are there any news regarding documentation? – Thomas B. Jan 25 '17 at 09:02
  • 2
    So have you had the time to write the tutorial ? :) – Muhammad Gelbana Mar 24 '17 at 08:59
  • 1
    Any chance to write up the doc? :) – Michael L Mar 28 '17 at 06:24