[Bug]: Missing embeddings in collections after a system reboot

[ymzayek]

What happened?

Hello, I'm working on a project where we use chromadb:0.5.11 as part of rag pipelines. We have succesfully used it to create collections and query them. We use our own embedder for the queries and chunks and do not rely on the chroma embedding method. We have just had an issue where it seemed that the embeddings in a collection got "deleted" or at least they are missing over the weekend after a reboot of the servers that we work on. To be clear, some query search tests on the collections before the weekend and system reboot clearly showed that the embeddings were well added to the collections at time of creation: a retrieval worked as it should to get the relevant documents. Now the same tests return an empty list of documents. I did some debugging by connecting to the chroma server with the chroma http client to maually check the collections and I see that the chunks and metadata still exists but the embeddings are empty. Has anyone seen this problem before? Any ideas about what could have happened or if it could be related to a system reboot? In the chroma logs all looks fine. I just see DEBUG: Starting component PersistentLocalHnswSegment sometimes but not sure that is related.

Possibly related to the following issues linked below but in our use case we do not delete any documents and then add new ones to the same collection. We just create one collection at a time which we then query without any further modifications in any of the data/documents in the collection.

#2512
#2062
#870

Versions

0.5.11

Relevant log output

No response

[HammadB]

Hi @ymzayek is it possible to share your db if its not sensitive data? Also happy to take them via discord/email if you don't want to post it here. That would help us debug on our end.

Otherwise, can we start by knowing what files are under /<chroma_path>/<collection_id>/

[tazarov]

@ymzayek, have a look at this issue #2922. I think the symptoms you are experiencing are similar.

[ymzayek]

@tazarov yes it seems to explain our problem so I think you can close this issue. Thanks!

[tazarov]

@ymzayek, excellent. However, you will have to recreate the missing embeddings. Let me know if you need help with that.

[ymzayek]

@tazarov yes we will handle that. Thanks a lot :)

[tazarov]

@ymzayek, can you confirm whether this solved your problem?

[ymzayek]

Apologies for the delay. For the moment we haven't upgraded (after a downgrade to 0.5.5). Are there any compatibility issues with the latest version and 0.5.5? But in any case, you can feel free to close this issue

[tazarov]

@ymzayek, there have been a number of bugfixes and improvements introduced in 0.5.13+. One notable thing that was fixed in 0.5.7 was a connection pool leak #2014 (long-standing problem in Chroma). From server compatibility I don't think you'll face any major hurdles. There's been some minor API changes - e.g. #2880.