qdrant

[xkzhud]

Context / Scenario

IKernelMemory kernelMemory = new KernelMemoryBuilder()
.WithCustomTextGenerator(new OneApiTextGenerator(options, _chatService))
.WithCustomEmbeddingGenerator(new OneApiTextEmbeddingGenerator(options, _chatService))
.WithCustomImageOcr(new HwsOcrEngine(_hwsService))
.WithSimpleFileStorage(new SimpleFileStorageConfig { StorageType = FileSystemTypes.Disk })
.WithQdrantMemoryDb(new QdrantConfig() { Endpoint = "http://10.187.80.248:6333", APIKey = "", DefaultIndex = DocumentMemorySettings.MemoryIndexName })
//.WithCustomTextPartitioningOptions(new TextPartitioningOptions
//{
// MaxTokensPerParagraph = 299,
// MaxTokensPerLine = 99,
// OverlappingTokens = 47,
//})
.Build();

Question

by using this config the vectordb index can not created and the data can not insert into the db . why?

### Tasks

[xkzhud]

and no bug message throw 。。

[xkzhud]

just message
text partitions not found, cannot generate embeddings, moving to next pipeline step.
embeddings not found, cannot save embeddings, moving to next pipeline step.

[xkzhud]

serverless mode process speed need upgrade , whatshould I do?

[dluc]

I tested example 003 using this initialization and everything worked fine:

IKernelMemory memory = new KernelMemoryBuilder()
    .WithCustomTextGenerator(new OpenAITextGenerator(openAIConfig, new DefaultGPTTokenizer(), (ILogger<OpenAITextGenerator>)null))
    .WithCustomEmbeddingGenerator(new OpenAITextEmbeddingGenerator(openAIConfig, new DefaultGPTTokenizer(), (ILogger<OpenAITextEmbeddingGenerator>)null))
    .WithCustomImageOcr(new AzureAIDocIntelEngine(azDocIntelConfig))
    .WithSimpleFileStorage(new SimpleFileStorageConfig { StorageType = FileSystemTypes.Disk })
    .WithQdrantMemoryDb(new QdrantConfig { Endpoint = "http://127.0.0.1:6333", APIKey = "" })
    .Build();

Could you provide more details about the code you're executing? Are you customizing the ingestion steps perhaps?

[xkzhud]

here is the code now work well .
KM config:
IKernelMemory kernelMemory = new KernelMemoryBuilder()
.WithCustomTextGenerator(new OneApiTextGenerator(options, _chatService))
.WithCustomEmbeddingGenerator(new OneApiTextEmbeddingGenerator(options, _chatService))
.WithCustomImageOcr(new HwsOcrEngine(_hwsService))
.WithCustomTextPartitioningOptions(textPartitioningOptions)
.WithQdrantMemoryDb(new QdrantConfig() { Endpoint = _options.Value.EndPoint, APIKey = _options.Value.ApiKey, DefaultIndex = DocumentMemorySettings.MemoryIndexName })
//.WithCustomTextPartitioningOptions(new TextPartitioningOptions
//{
// MaxTokensPerParagraph = 299,
// MaxTokensPerLine = 99,
// OverlappingTokens = 47,
//})
.Build<MemoryServerless>();

the upload function:
private async Task StoreDocumentAsync(IKernelMemory memoryClient, string documentId, string sessionId, string fileName, Stream fileContent, CancellationToken cancellationToken = default)
{
try
{
var uploadRequest =
new DocumentUploadRequest
{
DocumentId = documentId,
Files = new List<DocumentUploadRequest.UploadedFile> { new(fileName, fileContent) },
Index = DocumentMemorySettings.MemoryIndexName,
Steps = Microsoft.KernelMemory.Constants.DefaultPipeline.ToList()
};

      uploadRequest.Tags.Add(MemoryTags.TagSessionId, sessionId);
      uploadRequest.Tags.Add(MemoryTags.TagMemory, DocumentMemorySettings.DocumentMemoryName);
      var id = await memoryClient.ImportDocumentAsync(uploadRequest, cancellationToken);
      return await memoryClient.IsDocumentReadyAsync(id, DocumentMemorySettings.MemoryIndexName);
  }
  catch (Exception ex) when (ex is not SystemException)
  {
      return false;
  }

}

[xkzhud]

when upload ， the speed is slow. how can i optimize my code ? now the pipline executing in memory. when i use disk, the pipline excuteing with warning :text partitions not found, cannot generate embeddings, moving to next pipeline step.
embeddings not found, cannot save embeddings, moving to next pipeline step. so the embedding notexecuted and not saved into vectordb.

[xkzhud]

I tested example 003 using this initialization and everything worked fine:

IKernelMemory memory = new KernelMemoryBuilder()
    .WithCustomTextGenerator(new OpenAITextGenerator(openAIConfig, new DefaultGPTTokenizer(), (ILogger<OpenAITextGenerator>)null))
    .WithCustomEmbeddingGenerator(new OpenAITextEmbeddingGenerator(openAIConfig, new DefaultGPTTokenizer(), (ILogger<OpenAITextEmbeddingGenerator>)null))
    .WithCustomImageOcr(new AzureAIDocIntelEngine(azDocIntelConfig))
    .WithSimpleFileStorage(new SimpleFileStorageConfig { StorageType = FileSystemTypes.Disk })
    .WithQdrantMemoryDb(new QdrantConfig { Endpoint = "http://127.0.0.1:6333", APIKey = "" })
    .Build();

Could you provide more details about the code you're executing? Are you customizing the ingestion steps perhaps?

thanks for reply. see the code up

[xkzhud]

and MemoryServerless mode may used a lot. we may not use KM as a service deployed in server. we will use it as a component integrated into the app. haha