fix(arrow/ipc): allow reading custom metadata from record batch message - #554
Draft
rkyleg wants to merge 1 commit into
Draft
fix(arrow/ipc): allow reading custom metadata from record batch message#554rkyleg wants to merge 1 commit into
rkyleg wants to merge 1 commit into
Conversation
zeroshade
force-pushed
the
custom-record-batch-metadata
branch
from
January 14, 2026 18:57
8d2e418 to
e6e5587
Compare
zeroshade
marked this pull request as draft
August 10, 2026 16:36
Member
|
@rkyleg Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush. Note: This comment was drafted by an AI-assisted triage tool run by a maintainer, and may contain mistakes. Once you have addressed the points above, an Apache Arrow Go maintainer — a real person — will take the next look at your PR. If anything here looks wrong, say so on the PR and a maintainer will follow up. See CONTRIBUTING.md for the project's contribution conventions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I could not find a method to read the custom metadata out of the RecordBatch message, only the metadata from the schema, so I added an option to read custom metadata from record batch message in ipc reader. This was the api I came up with. I tried to follow the existing methodology of updating the current record batch in Read/Next. Let me know if any api changes should be made or if I am completely missing this functionality somewhere.
Rationale for this change
Allow reading custom metadata from record batch message similar to the iter_batches_with_custom_metadata function in the pyarrow library.
What changes are included in this PR?
Add an ipc.Option (WithCustomRecordBatchMetadata)
Add meta field in ipc.Reader struct to hold metadata
Check WithCustomRecordBatchMetadata option in next() method on Reader and, if true, read metadata from the
Are these changes tested?
Existing tests pass; however, I am not sure how to use the current test setup to test these changes because there isn't an api (that I can tell) to write this custom metadata to an ipc either. I will look into that next.
Also wasn't sure where to upload example arrow stream files.
I did create an example of using these changes here.
If you have any pointers here, I would be happy to write some tests.
Are there any user-facing changes?
No