MarkdownSplitter: Smarter Table Splitting with Header Preservation

[hburrichter]

Hello @benbrandt,

First, thank you for your work on the text-splitter library!

I would like to propose a feature enhancement for smarter table splitting that preserves the header in all chunks.

Feature Request:

• Implement a smarter table splitting feature that automatically includes the header row in each split section of a table.
• Ensure that the markdown formatting is preserved across all splits.

Use Case:

This feature would be useful for markdown documents with large tables to maintain the readability and formatting consistency in each table chunk.

Example:

Consider a markdown table that is too large to fit in a single section:

| Header 1 | Header 2 | Header 3 |
|----------|----------|----------|
| Row 1    | Data     | More Data|
| Row 2    | Data     | More Data|
| Row 3    | Data     | More Data|
| Row 4    | Data     | More Data|

If the table needs to be split after the second row, the desired output would be:

Split 1:

| Header 1 | Header 2 | Header 3 |    
|----------|----------|----------|
| Row 1    | Data     | More Data|
| Row 2    | Data     | More Data|

Split 2:

| Header 1 | Header 2 | Header 3 |    <!-- currently missing -->
|----------|----------|----------|    <!-- currently missing -->
| Row 3    | Data     | More Data|
| Row 4    | Data     | More Data|

Thank you for considering this feature request!