Is there a way to upload a deep learning model to a PostgreSQL database?

Question

I have deep learning models (tensorflow in hdf5 format), which I want to upload to a PostgreSQL database. A single model may be up to 500 MBs, and the models need to be updated and uploaded/downloaded from the database frequently (e.g., once every 30 mins).

I'm a begineer to PostgreSQL, so any information/receipes to start with would be appreciated.

You can store any type of content in a bytea column, including hdf5 — Frank Heikens, Feb 17 '23 at 07:50
@wtz can you please give more details? what do you mean by saving on the file system. — Mr. Gulliver, Feb 17 '23 at 12:56
@LaurenzAlbe the models are in hdf5 format, including weights of the model (usually float64 data type) — Mr. Gulliver, Feb 17 '23 at 12:57
@FrankHeikens Thanks for the suggestion! Will search for it, would appreciate it if you have any examples — Mr. Gulliver, Feb 17 '23 at 12:58
Here a python example for storing a blob/bytea: https://www.postgresqltutorial.com/postgresql-python/blob/ — Frank Heikens, Feb 17 '23 at 13:33
@LaurenzAlbe The model can be decomposed into its constituent layers and each can be saved as a numpy file (npy, npz). Can the each npy or npz file be saved/fetched into/from the database separately? Thanks a lot for your help! — Mr. Gulliver, Feb 27 '23 at 15:13
You can save a file to the database as `text` (if it contains text) or `bytea` (if it is binary). Usually it is smarter to store larger files outside of a database. — Laurenz Albe, Feb 27 '23 at 15:30

wtz · Answer 1 · 2023-02-18T02:03:01.183

0

Storing large binary data in an relational DB is possible but unwise. Also, you have to eventually copy your model from the DB to the disk for TensorFlow to load anyway. Why not store your model directly on the disk?

edited Feb 18 '23 at 02:03

answered Feb 17 '23 at 16:17

wtz

426
4
15

The data which the model is trained on is separated into two different servers. No direct connection exists between the two servers, but both can access the database. So both ends will load the model from the database and train (fine-tune) it on the local data, then reupload it to the database.. sorry if this was a bit confusing, but data has high privacy, cannot be shared outside the servers.. – Mr. Gulliver Feb 27 '23 at 15:16

score 0 · Answer 2 · answered Feb 17 '23 at 16:20

0

Relation DB is a wrong tool for this. Store the link to an S3 bucket for example. Or your own private cloud if the model is private.

answered Feb 17 '23 at 16:20

Jordan Walker

13
2

Is there a way to upload a deep learning model to a PostgreSQL database?

2 Answers2