MDEV-39762: Slave Overflow on Malformed Query_compressed_log_event - #5517
Open
bnestere wants to merge 2 commits into
Open
MDEV-39762: Slave Overflow on Malformed Query_compressed_log_event#5517bnestere wants to merge 2 commits into
bnestere wants to merge 2 commits into
Conversation
A replica can crash when a compressed event from its master carries a corrupted uncompressed length. A compressed Query event whose length is near 4GB overwrites the IO thread's stack with the event's content. A compressed rows event of the same shape makes the replica ask for an allocation of 4GB. A debug build that gets that memory then fails an assertion in binlog_buf_uncompress(). On a build where a ulong is 32 bits wide, both events overwrite the IO thread's stack. The IO thread uncompresses a compressed event before writing the event to the relay log, and offers the uncompress function 4096 bytes of its own stack to hold the result. The function computes how large the event will be once uncompressed, and that size decides where the uncompressed event goes. An event that fits in the stack buffer is uncompressed there. A larger event is uncompressed into an allocation the function makes at that size. With either buffer, the function tells zlib that the room available is the uncompressed length the event declared. query_event_uncompress() never bounded the length read out of the event. The function added the header length to that value, rounded the sum up for alignment, and cast the result to uint32. The cast dropped the high bits of a sum above 4GB, so a length near 4GB produced a size of a few dozen bytes. That size fit the stack buffer, so content of any size went onto the IO thread's stack. row_log_event_uncompress() computes that sum in a ulong, which wraps where a ulong is 32 bits wide. The Query_compressed_log_event constructor and Rows_log_event::uncompress_buf() size their allocations from the same unbounded length. Reject an uncompressed length above MAX_MAX_ALLOWED_PACKET at every point the length is read, before any size is computed from it. No master writes a larger length, because max_allowed_packet is capped at 1GB. query_event_uncompress() also keeps that size in a size_t, so the allocation is made from the whole computed value. Both functions return the error code for the IO thread to report, so a length over the limit stops the IO thread with ER_TOO_BIG_FOR_UNCOMPRESS rather than the generic uncompress error. A compressed event with a corrupted uncompressed length now stops the replica's IO thread with an error instead of crashing the replica. Alexandra Raveala wrote the original patch and its first regression test, contributed through PR 5413. Brandon Nesterenko saw the work through for the 10.6 release, replacing the test with one that makes the master write the corrupt length and reworking how the IO thread reports the error. Co-authored-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com> Reviewed-by: TODO Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Member
|
Brandon Nesterenko ***@***.***> writes:
A replica can crash when a compressed event from its master carries a
corrupted uncompressed length. A compressed Query event whose length
Thanks Brandon, looks fine to me.
Reviewed-by: Kristian Nielsen ***@***.***>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A replica can crash when a compressed event from its master carries a
corrupted uncompressed length. A compressed Query event whose length
is near 4GB overwrites the IO thread's stack with the event's content.
A compressed rows event of the same shape makes the replica ask for an
allocation of 4GB. A debug build that gets that memory then fails an
assertion in binlog_buf_uncompress(). On a build where a ulong is 32
bits wide, both events overwrite the IO thread's stack.
The IO thread uncompresses a compressed event before writing the event
to the relay log, and offers the uncompress function 4096 bytes of its
own stack to hold the result. The function computes how large the event
will be once uncompressed, and that size decides where the uncompressed
event goes. An event that fits in the stack buffer is uncompressed
there. A larger event is uncompressed into an allocation the function
makes at that size. With either buffer, the function tells zlib that
the room available is the uncompressed length the event declared.
query_event_uncompress() never bounded the length read out of the
event. The function added the header length to that value, rounded the
sum up for alignment, and cast the result to uint32. The cast dropped
the high bits of a sum above 4GB, so a length near 4GB produced a size
of a few dozen bytes. That size fit the stack buffer, so content of any
size went onto the IO thread's stack. row_log_event_uncompress()
computes that sum in a ulong, which wraps where a ulong is 32 bits
wide. The Query_compressed_log_event constructor and
Rows_log_event::uncompress_buf() size their allocations from the same
unbounded length.
Reject an uncompressed length above MAX_MAX_ALLOWED_PACKET at every
point the length is read, before any size is computed from it. No
master writes a larger length, because max_allowed_packet is capped at
1GB. query_event_uncompress() also keeps that size in a size_t, so the
allocation is made from the whole computed value. Both functions return
the error code for the IO thread to report, so a length over the limit
stops the IO thread with ER_TOO_BIG_FOR_UNCOMPRESS rather than the
generic uncompress error. A compressed event with a corrupted
uncompressed length now stops the replica's IO thread with an error
instead of crashing the replica.
Alexandra Raveala wrote the original patch and its first regression
test, contributed through PR 5413. Brandon Nesterenko saw the work
through for the 10.6 release, replacing the test with one that makes
the master write the corrupt length and reworking how the IO thread
reports the error.
This PR is organized in two commits. The first is the regression test and
debug injections; the second is the fix and .result file.