-1

I have a 2048 dimension numpy array. I want to store in postgresql and find.

I tried cube data type (I increased the size to 2048 dimension) but I get "row is too big" error because size is big

I tried to keep it as text but I can't get cosine similarity.

How do I store and find this data? It can be in a different database or method. I'm open to all kinds of suggestions

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Ciii
  • 63
  • 1
  • 11

1 Answers1

0

You could save it in a table like this:

VectorId   Index    value
1          0        0.5
1          1        2.5
...        ...      ...
1          2048     3.0
2          0        1.5
2          1        4.5
...        ...      ...
2          2048     0.0
3          0        1.3
...        ...      ...

Thats a lot of "unnecessary" metadata though. But if you don't have too much vectors, it shouldn't be a problem.

Here is a post where the data is saved the same way and the question is about calculating the cosine similarity with sql only.

SQL Computation of Cosine Similarity

By the way: You have a one-dimensional array/vecor with a length of 2048, that was a little unclear in the question.

Florian H
  • 3,052
  • 2
  • 14
  • 25
  • I guess I didn't quite know what I wanted. I have a 2 numpy array (2048 dimension). [2.5203197,3.3983448,0.0014629656,2.1648185,0.16088574,0.0,0.27310637,1.2325579] and [0.62217987,0.0,0.0,0.34341377,0.0,0.0,5.2953234,0.01631545,0.07210871,0.0,0.68821776]. I want to store database and find cosine similarity on database with query – Ciii Aug 23 '19 at 06:50
  • can you post the shape of your numpy array? does it have a shape like (2048) or (a1, a2, ..., a2048)? – Florian H Aug 23 '19 at 08:40
  • like (a1, a2, a3 .... a2048) all of them is float values. And i wanna store them and find them (with cosine similarity) – Ciii Aug 24 '19 at 10:07