How to save/load faiss KMeans for later inference

Question

I have successfully clustered a bunch of vectors using the faiss kmeans. But now I am not able to store the model and load it later for inference.

clustering = faiss.Kmeans(candles.shape[1], k=clusters, niter=epochs, gpu=gpu, verbose=True)
clustering.train(X)
cluster_index = clustering.index

# failed with "don't know how to serialize this type of index"
faiss.write_index(cluster_index, f"{out_file}.faiss")
model2 = faiss.read_index(f"{out_file}.faiss")

model2.search(x, 1)

score 0 · Answer 1 · answered Mar 23 '23 at 09:28

Looks like you can store the centeroids as a numpy/pandas and then re-create the index:

pd.DataFrame(model.centroids).parquet(filename)
df_centroids = pd.read_parquet(filename)
cluster_index = faiss.IndexFlatL2(df_centroids.shape[1])
cluster_index.add(df_centroids.values.copy(order='C'))

However, this feels a little bit wrong since you can not know what distance measure was used for training.

How to save/load faiss KMeans for later inference

1 Answers1