0

For datajoint-matlab, #327 seems to indicate that File external storage (#143, PR #197) should be implemented in the current version. I can create a table with the datatype filepath@data, after defining the store 'data', but I get an error on insert.

Error using dj.Relvar/insert/makePlaceholder (line 244)
The field `fref` with datatype `filepath@data` is not yet supported.

Error in dj.Relvar/insert (line 334)
                    [v, placeholder] = makePlaceholder(i,
                    tuple.(header.attributes(i).name));

Is this still not implemented, or is the error-checking here just stopping me from using it? Happy to provide more details on the test if needed.

Gary Yellen
  • 1

1 Answers1

1

Maintainer for DataJoint here. Looks like there is a bit of confusion so let's see if I can help to bring some clarity. Hope to use this discussion as a resource to improve the documentation.

DataJoint provides a few DataJoint-only datatypes. Of these types, we identify the ones associated with external storage by embedding an @ symbol. We classify each part of the type as <datatype>@<store>. Essentially with these types, information (i.e. datatype) is stored remotely in an object store (i.e. a store) with proper reference links within the relational database and client configuration for access.

For datatype, there are currently 3 options:

  • blob: Equivalent to a blob type but for external stores. Currently, this type is supported both in datajoint-python and most recently in datajoint-matlab.
  • attach: A special type that captures file content as binary information but does not preserve any path information. Currently, this type is supported only in datajoint-python. Documentation is available for this type in File Attachment Datatype section.
  • filepath: A special type that captures file content as binary information and includes path (along with file name) detail. Currently, this type is in preview in datajoint-python requiring it to be enabled. Documentation is available for this type in Filepath Datatype section. See note in documentation to enable it.

For store, there is an External Store section in the documentation. Multiple stores can be configured as a mapping located under the stores key in dj.config. For MATLAB, see help('dj.config') for examples and in Python, refer to the docs for attach and filepath above.

Stores currently support 2 protocols:

Issue #143 and PR's #197, #327 that you've mentioned refer to efforts in implementing file, s3 stores for the blob datatype in DataJoint MATLAB. The error you've experienced is expected and a simple placeholder until we support the other 2 datatypes in DataJoint MATLAB.

Raphael Guzman
  • 220
  • 2
  • 7