Compactor: writing block: closing index writer: postings offset table length/crc32

[cabrinha]

Describe the bug

Compactors have been throwing an odd error, not described in the runbooks

write error: length size exceeds 4 bytes: 4337654064

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (latest)
2. Perform Operations (Read/Write/Others)

Expected behavior

I expect compactor to not throw errors

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

I would like to see an example of scaling compactors to accommodate a large number of series/blocks. Split and merge sharding configuration isn't clear.

[pstibrany]

Hello,

as you have discovered, you will need to enable sharding of series into multiple blocks in compactor (we also call this "splitting").

This sharding is applied to compaction jobs at smallest compaction range (2h) to shard series into multiple target blocks. By default there's no sharding of series done, but that can result in a large blocks. You need to configure sharding, you can start with 2 shard, and increase that if you get failures again. Command line option for this is compactor.split-and-merge-shards or you can use per-tenant override in runtime-config compactor_split_and_merge_shards.

As mentioned, sharding is only applied at smallest compaction range (2h blocks). If you already have larger blocks that are failing to merge because result is too big, compactor cannot merge such blocks. The only way to avoid compactor complaining is to use markblocks tool (can be downloaded from https://github.com/grafana/mimir/releases), and mark block for non-compaction. Then compactor will not try to compact the blocks anymore, and will move on to other blocks.

[cabrinha]

Thanks for the response! Are there any other ways to optimize the compactor? Is the process more cpu, memory or disk intensive?

[pstibrany]

Compactor is by default configured with -compactor.compaction-concurrency=1, and compaction is mostly single-core process, so 1 cpu core is enough. I'd only add more memory if compactors OOMed. Same for disk -- if compaction jobs are not failing with current disk size, then compactor has enough space.

You can however run more compactors if single compactor is lagging behind (symptoms like unbounded increase of blocks, slow queries). Compactors will then coordinate and shard the jobs between them.

[cabrinha]

I'm up to 128 shards and in 16 groups now but still getting the error. Should I increase shards or groups?