Automatic and continuous indexing of existing data sources

[rosieks]

Context / Scenario

Common use case of using AI is to provide answer based on given existing knowledge base (it might be some company wiki, git repository, system database)

The problem

Right now the only way to fill kernel memory is by importing all documents one by one.

Proposed solution

It would be nice to have some mechanism to synchronize kernel memory with external data source (eg. azure blob storage, git repository, wiki, sharepoint). The idea is to provide abstraction so people could use it to implement their own data sources. Concepts that it has to provide is:

• Initial data feed
• Data synchronization (polling or listening - depending of data source possibilities)

Importance

None

[koteus]

Right now for this use-case I had to add some custom logic to my project, which works like this: when user adds new knowledge from a URL and user checks the checkbox to keep it up-to-date I create a record in my crawler_tasks table. Then I have this crawler service that runs periodically and fetches the web page and updates the knowledge. In some cases the page requires authentication or maybe it is a link to Google Docs etc, to address that I have "adapters" on my crawler, the crawler decides to use an adapter based on url pattern (it extracts domain and checks if there is an adapter for it, otherwise it falls back to regular webpage adapter). An adapter is a class that defines how to fetch this kind of resource.

We needed this feature in my company because we have some wiki-style resources which are updated regularly and we wanted Kernel Memory to be in sync with the latest changes. So now when we provide a parent url of our wiki, it can also check all children links recursively and if they are of the same origin it will add them as documents, but we also have to track what was a parent document for each of them. When user deletes parent we also want to delete all children.

It would be great if we integrate these features to Kernel Memory itself when it runs as a service.
What do you think @dluc does it align with the vision of Kernel Memory?

[dluc]

If you use the asynchronous version of KM, with queues, you could use a custom handler to do a lot of that.

• Create a custom handler, and assign it a name, e.g. processNewKnowlege
• Enable the custom handler in KM service
• When a user adds new knowledge call MemoryWebClient.ImportTextAsync(...) (or the corresponding web API endpoint), passing a JSON with data for your custom handler, and set steps: ["processNewKnowlege"]. This will enqueue an operation and your handler will start working on it.
• In your custom handler, load the JSON file, which will contain data that you need, e.g. the URL to process, what type of auth is needed, etc. This handler can do everything you want, including calling KM to store data you prepared with custom logic