0

I'm trying to set up an external store to work with my pipeline. I am self hosting my database on a small server which my lab connect to. This is currently working well and we can all insert and query the database. However, we have some large arrays that would be too big to store in the database so we want to use DataJoint's external store feature to store these.

Currently, when I assign the location in dj.config['stores'], it assigns to a location on my local machine. I would like to assign this to a location on the server (the same server as the database), so that the external data is stored centrally and can be accessed by all members of the lab. Can anyone tell me how to do this? Many thanks.

rbedford
  • 5
  • 2

1 Answers1

0

I'd expect you are trying this on your local and it works, but you want to know how to point the location to your on-premise server in order to do the same?

dj.config['stores'] = {
  'data': {
    'protocol': 'file',
    'location': '/data',
    'stage': '/data'
  }
}

If I'm not wrong, in order to achieve that, you need to either setup your own S3 store on that server or you make that server as a NFS(Network File System) that you can mount the server's file system through the network on each user's work machine. The NFS solution has a network boundary limitation that people can only work with it in your lab network(like you can't mount it at home)... However, the S3 store solution doesn't have that limitation.

To correct myself, technically maybe you can connect with your lab NFS from home through VPN, but I've never done that before. I do recommend you go with the S3 store solution, it has better scalability and file management, just in case one day you want to migrate to the cloud.

Drew Yang
  • 76
  • 3