I switched from the MemoryStore to this sqlite implementation because I was running out of memory with a large number of queued tasks in batch mode. I am finding that the _writeQueue length is growing continuously (only dropping a little on occasion). This eventually leads to an out of memory error. Any tips on how to remedy this? I can throttle things on the job generation side, but the motivating reason for adding the work queue was to avoid that.
I switched from the MemoryStore to this sqlite implementation because I was running out of memory with a large number of queued tasks in batch mode. I am finding that the _writeQueue length is growing continuously (only dropping a little on occasion). This eventually leads to an out of memory error. Any tips on how to remedy this? I can throttle things on the job generation side, but the motivating reason for adding the work queue was to avoid that.