Skip to content

fix(arrow/ipc): allow reading custom metadata from record batch message - #554

Draft
rkyleg wants to merge 1 commit into
apache:mainfrom
rkyleg:custom-record-batch-metadata
Draft

fix(arrow/ipc): allow reading custom metadata from record batch message#554
rkyleg wants to merge 1 commit into
apache:mainfrom
rkyleg:custom-record-batch-metadata

Conversation

@rkyleg

@rkyleg rkyleg commented Oct 30, 2025

Copy link
Copy Markdown

I could not find a method to read the custom metadata out of the RecordBatch message, only the metadata from the schema, so I added an option to read custom metadata from record batch message in ipc reader. This was the api I came up with. I tried to follow the existing methodology of updating the current record batch in Read/Next. Let me know if any api changes should be made or if I am completely missing this functionality somewhere.

Rationale for this change

Allow reading custom metadata from record batch message similar to the iter_batches_with_custom_metadata function in the pyarrow library.

What changes are included in this PR?

Add an ipc.Option (WithCustomRecordBatchMetadata)
Add meta field in ipc.Reader struct to hold metadata
Check WithCustomRecordBatchMetadata option in next() method on Reader and, if true, read metadata from the

Are these changes tested?

Existing tests pass; however, I am not sure how to use the current test setup to test these changes because there isn't an api (that I can tell) to write this custom metadata to an ipc either. I will look into that next.
Also wasn't sure where to upload example arrow stream files.
I did create an example of using these changes here.
If you have any pointers here, I would be happy to write some tests.

Are there any user-facing changes?

No

@rkyleg
rkyleg requested a review from zeroshade as a code owner October 30, 2025 02:07
@zeroshade zeroshade changed the title Read custom metadata from record batch message fix(arrow/ipc): allow reading custom metadata from record batch message Jan 14, 2026
@zeroshade
zeroshade force-pushed the custom-record-batch-metadata branch from 8d2e418 to e6e5587 Compare January 14, 2026 18:57
@zeroshade
zeroshade marked this pull request as draft August 10, 2026 16:36
@zeroshade

Copy link
Copy Markdown
Member

@rkyleg Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

  • Merge conflicts — this branch conflicts with main. See docs.

Note: Your branch is 390 commits behind main. Please rebase and push again to get up-to-date CI results.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.


Note: This comment was drafted by an AI-assisted triage tool run by a maintainer, and may contain mistakes. Once you have addressed the points above, an Apache Arrow Go maintainer — a real person — will take the next look at your PR. If anything here looks wrong, say so on the PR and a maintainer will follow up. See CONTRIBUTING.md for the project's contribution conventions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants