0

I have a very large Weaviate vector storage class (700,000 records) in which I pass my own custom vectors. I’m trying to get distances against a vector I pass as below. The vector is actually a match to one of the records, so I know the top hit should be the record with the identical vector (distance very close to 0). However, when I ask for top hits, the “closest” record is returning a distance of around 0.10, and this record is definitely not the record that matches my query vector perfectly (node_type="type1" instead of "type2").

# NOTE: mean_emb is a numpy array that matches a record pushed to the MyClass weaviate class.
# This theoretically should return distances from all 700k records to specified vector, since "distance" = 1.0, but I get why it wouldn't computationally
result = (client.query.get("MyClass", ["message", "node_type", "my_id", "timestamp"])
          .with_near_vector({"vector": mean_emb.tolist(), "distance": 1.0})
          .with_additional(["vector", "distance"]).do())
result = result["data"]["Get"]["MyClass"]
print(len(result))  # only 11,100 distances are returned

It looks like with_offset() doesn’t like it when the offset is >100,000.

I have tried pagination using with_after() but with_after doesn’t support queries with with_near_vector(), and I have also tried with_offset() + with_limit(), but this is terribly slow. Is there a workaround / what am I doing wrong here / how to query my class so that my top N query includes the true record match (distance close to 0)?

To prove there is in fact a record with distance ~0.000. Here’s the query that highlights the records that matches the vector:

where_filter = {"path": ["node_type"], "operator": "Equal", "valueText": "type2"}
result = (client.query.get("MyClass", ["message", "node_type", "my_id", "timestamp"])
          .with_near_vector({"vector": mean_emb.tolist()})
          .with_additional(["distance", "id"]).with_where(where_filter).do())
print(result)

Gives me this (I’ve changed the values of some of the record meta-data to protect data):

{'data': {'Get': {'MyClass': [{'_additional': {'distance': -1.9073486e-06,
      'id': 'fdb00f95-2c07-462c-84cd-9380c6777801'},
     'my_id': 'Record that matches the vector passed',
     'message': None,
     'node_type': 'type2',
     'timestamp': None},
    {'_additional': {'distance': 0.6122676,
      'id': '0deb152a-eef0-485c-ad6e-c9e29f9a3915'},
     'my_id': 'Another type2 record that doesn't match vector passed',
     'message': None,
     'node_type': 'type2',
     'timestamp': None}]}}}
lrthistlethwaite
  • 494
  • 2
  • 6
  • 23

1 Answers1

0

If you insist on having the the exact match appear at the top, then, you need to tweak the indexing algorithm (HNSW) to increase recall. Try increasing the value of the ef paramater.

hsm207
  • 471
  • 2
  • 4
  • I haven't yet been able to verify that this solves my problem, but you've definitely unlocked a part of the documentation I didn't know to look at, so thank you so much! I tried ef = 100 but top hits still were not there. I noticed that the doc of interest had two properties that were being vectorized, whereas docs of type 1 only had one property that was vectorized. Would that affect the rankings? I'll have to repush and test but it'll take a few days to verify, etc. – lrthistlethwaite Jul 25 '23 at 19:47
  • ef=100 is too small in opinion, considering the number of records you have. Have a look here to understand what ef controls: https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md – hsm207 Jul 26 '23 at 05:11
  • Great references / links! Thank you for highlighting this functionality for me! – lrthistlethwaite Jul 26 '23 at 14:06