Support IPC Message custom_metadata in ArrowStreamWriter and ArrowStreamReader (rebased continuation of #283) - #424
Conversation
The Arrow IPC format supports custom_metadata on each Message (RecordBatch), but the C# implementation currently ignores it on read. This adds a LastBatchCustomMetadata property to ArrowStreamReader that exposes the key-value pairs from the most recently read batch's Message. This is the read-side counterpart to pyarrow's read_next_batch_with_custom_metadata() and enables use cases like RPC frameworks that embed method routing or log metadata in per-batch custom_metadata fields. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds WriteRecordBatch(batch, customMetadata) and its async counterpart to ArrowStreamWriter, allowing callers to attach per-message custom_metadata key-value pairs when writing IPC streams. The Arrow IPC flatbuf Message already defines a custom_metadata field, and pyarrow supports writing it via write_batch(batch, custom_metadata). This brings the C# writer to parity. Includes round-trip tests verifying custom_metadata survives write → read through ArrowStreamWriter/ArrowStreamReader. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- C# writes IPC stream with custom_metadata → Python reads via read_next_batch_with_custom_metadata() - Python writes IPC stream with custom_metadata → C# reads via LastBatchCustomMetadata
There was a problem hiding this comment.
Pull request overview
Adds support for Arrow IPC Message.custom_metadata in the .NET IPC stream reader/writer APIs, plus tests to validate same-language and Python round-trips. This brings Apache.Arrow .NET closer to parity with other Arrow implementations (e.g., PyArrow) for per-record-batch message metadata.
Changes:
- Add
ArrowStreamWriter.WriteRecordBatch(..., customMetadata)(sync + async) to emit IPCMessage.custom_metadataon record batch messages. - Add
ArrowStreamReader.LastBatchCustomMetadatato expose custom metadata from the most recently read record batch message. - Add .NET round-trip tests and Python interop tests (Python.NET + PyArrow) for custom metadata.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Apache.Arrow.Tests/CustomMetadataPythonTests.cs | Adds cross-language Python interop tests for reading/writing IPC custom metadata. |
| test/Apache.Arrow.Tests/ArrowStreamWriterTests.cs | Adds in-process C# writer/reader round-trip tests for custom metadata behavior. |
| src/Apache.Arrow/Ipc/ArrowStreamWriter.cs | Serializes Message.custom_metadata when writing record batches; adds overloads and message-writing updates. |
| src/Apache.Arrow/Ipc/ArrowStreamReader.cs | Exposes LastBatchCustomMetadata on the public stream reader API. |
| src/Apache.Arrow/Ipc/ArrowReaderImplementation.cs | Parses Message.custom_metadata and stores it for access after reading a record batch. |
Suppressed comments (1)
test/Apache.Arrow.Tests/CustomMetadataPythonTests.cs:71
Dispose()unconditionally callsPythonEngine.Shutdown(), even when initialization was skipped/failed. This can also interfere with other Python.NET tests running in the same process. The existingPythonNetFixtureguards shutdown behindInitialized.
public void Dispose()
{
PythonEngine.Shutdown();
}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| long metadataLength = await WriteMessageAsync(Flatbuf.MessageHeader.RecordBatch, | ||
| recordBatchOffset, recordBatchBuilder.TotalLength, | ||
| customMetadataVectorOffset, | ||
| cancellationToken).ConfigureAwait(false); |
There was a problem hiding this comment.
Fixed in 0177e8b: FlightDataStream now overrides the customMetadata-aware WriteMessageAsync overload (the one WriteRecordBatchInternalAsync actually calls) instead of the old 4-arg one, so Flight record batch writes are no longer silently routed through the base implementation.
| // Build custom metadata for the Message if provided | ||
| VectorOffset customMetadataVectorOffset = default; | ||
| if (customMetadata != null && customMetadata.Count > 0) | ||
| { | ||
| Offset<Flatbuf.KeyValue>[] metadataOffsets = GetMetadataOffsets(customMetadata); | ||
| customMetadataVectorOffset = Flatbuf.Message.CreateCustomMetadataVector(Builder, metadataOffsets); |
There was a problem hiding this comment.
Fixed in 0177e8b: added a ValidateCustomMetadata check that throws a clear ArgumentException for null keys/values before building the FlatBuffer offsets.
| /// <summary> | ||
| /// Custom metadata from the most recently read RecordBatch Message. | ||
| /// Updated after each call to ReadNextRecordBatch/ReadNextRecordBatchAsync. | ||
| /// Returns null if the last batch had no custom metadata. | ||
| /// </summary> |
There was a problem hiding this comment.
Fixed in 0177e8b: reworded the doc comment to describe the actual semantics — the property is left unchanged (not cleared) when a read call returns null, e.g. at end of stream.
- FlightDataStream: override the customMetadata-aware WriteMessageAsync
overload (the one WriteRecordBatchInternalAsync now actually calls)
instead of the old 4-arg overload, so Flight writes aren't silently
routed through the base implementation and don't bypass DataHeader
capture.
- ArrowStreamWriter: validate that caller-supplied custom metadata has
no null keys/values before building FlatBuffer offsets, so failures
are reported as a clear ArgumentException rather than an opaque
FlatBufferBuilder exception.
- ArrowStreamReader: correct the LastBatchCustomMetadata XML doc to
describe its actual update semantics (it's left unchanged when a
read call returns null, e.g. at end of stream).
- CustomMetadataPythonTests: use the repo's shared PythonNetFixture +
[Collection("PythonNet")] instead of a private per-class Python.NET
init/shutdown, avoiding double-Initialize/premature-Shutdown races
with other Python.NET tests; this also fixes the missing Py.GIL()
guard around the Windows sys.path append, since the shared fixture
already wraps that in using (Py.GIL()).
Verified with:
- dotnet build Apache.Arrow.sln — 0 warnings/errors
- dotnet test test/Apache.Arrow.Tests/Apache.Arrow.Tests.csproj —
1876 passed, 30 skipped, 0 failed
- dotnet format Apache.Arrow.sln --exclude src/Apache.Arrow/Flatbuf/FlatBuffers/ --verify-no-changes — clean
| /// <returns> | ||
| /// The number of bytes written to the stream. | ||
| /// </returns> | ||
| private protected virtual ValueTask<long> WriteMessageAsync<T>( |
There was a problem hiding this comment.
Consider making this non-virtual or removing it entirely and forcing use of the signature with the customMetadataOffset (which is what WriteMessage does).
|
|
||
| public virtual void WriteRecordBatch(RecordBatch recordBatch, IReadOnlyDictionary<string, string> customMetadata) | ||
| { | ||
| WriteRecordBatchInternal(recordBatch, customMetadata); |
There was a problem hiding this comment.
Based on my previous analysis of the original PR, this overload breaks ArrowFileWriter because it would skip the call to WriteStart which is performed in that class's WriteRecordBatch/WriteRecordBatchAsync overrides.
| FinishedWritingRecordBatch(bufferLength, metadataLength); | ||
| } | ||
|
|
||
| private protected Task WriteRecordBatchInternalAsync(RecordBatch recordBatch, |
There was a problem hiding this comment.
My suspicion is that it's better not to have this overload. We don't need it for backwards-compatibility and removing it might avoid an error in a derived class.
Continuation of #283, rebased onto current
main(the original PR had drifted ~76 commits behind and could no longer be evaluated for mergeability).Original work by @cmettler in #283, closing #282. This PR carries those three commits forward unchanged (same authorship) on top of the current
main, which in the interim gained unrelated IPC changes (RunEndEncodedArray, LargeListView, IPC buffer bounds validation, etc.) that touch the same files.What this adds
ArrowStreamReader.LastBatchCustomMetadata: exposes the IPCMessage.custom_metadatakey/value pairs for the most recently read batch — the read-side counterpart to pyarrow'sread_next_batch_with_custom_metadata().ArrowStreamWriter.WriteRecordBatch(batch, customMetadata)(and async counterpart): lets callers attach per-messagecustom_metadatawhen writing IPC streams, matching pyarrow'swrite_batch(batch, custom_metadata).PYTHONNET_PYDLLis set, consistent with the existingCDataSchemaPythonTestpattern in this repo).Verification on this rebase
dotnet build Apache.Arrow.sln— succeeds, 0 warnings/errors.dotnet test test/Apache.Arrow.Tests/Apache.Arrow.Tests.csproj— full suite: 1876 passed, 30 skipped (pre-existing Python-dependent tests, unrelated to this change), 0 failed.dotnet format Apache.Arrow.sln --exclude src/Apache.Arrow/Flatbuf/FlatBuffers/ --verify-no-changes— clean.Closes #282
Supersedes #283
🤖 Generated with Claude Code