Additional fields in index to support selecting newer and/or boosted content

[chaelli]

I wonder if it would make sense to have additional fields in the index. I have three usecases (similar) for this:

• having a "last updated" date to be able to select / prefer newer content
• having a "boost" field that could be used to boost specific content
• having a "title" field would allow for the LLM to directly include sources with title and url

Or are you all selecting just "everything" that matches?

[dluc]

Right there's a lot in this area to research. One brute-force approach would be to label records with a timestamp, but it would be impossible to scale. KM arch doesn't take into account time. There are other things you could consider:

• During ingestion, search for similar records and merge. This would allow also to resolve contraddictions.
• During ingestion, assign a time label (e.g. day 20240617). When running RAG prompt, tell the model to give priority to records with a recent time label.
• During ingestion, find similar memories and increase weight, eg. using a metadata field. When running RAG tell the model to consider the weight.

I'm sure there's more and probably better ways, hopefully the solution allows to explore these and more ideas

[chaelli]

thanks for the input. one disadvantage of the need to be flexible ;) I'll try to play around with these ideas

[dluc]

by the way, this PR just merged might give some extra tools to leverage #673 e.g. the ability to include in the RAG prompt tags that you can ask the model to consider

[chaelli]

this is very cool! Thanks - I was already looking into options for dynamic prompts - this will make this simpler!