Bound total memory used by HiveSplitSource #9119
Conversation
There was a problem hiding this comment.
The next commit introduces side effect to the for-loop
There was a problem hiding this comment.
Why times two? There is only one String in the class.
There was a problem hiding this comment.
Just add
.map(Object::toString)There was a problem hiding this comment.
This is long and hard to read, add parentheses
There was a problem hiding this comment.
I disagree on this one. I find it harder to read with parenthesis.
I will address all other comments. I noticed the build failure and fixed them myself. Apparently, I forgot to push it.
There was a problem hiding this comment.
Using Ints.saturatedCast() might be better here. It means the memory limit will be lower than the very large limit that was specified.
There was a problem hiding this comment.
This can overflow if the limit is close to MAX_INT, and/or if there are many threads incrementing at once
There was a problem hiding this comment.
We should probably make this a long, just to be safe
There was a problem hiding this comment.
format(
"Split buffering for %s.%s exceeded memory limit (%s). %s splits are buffered.",
..., succinctBytes(maxOutstandingSplitsBytes), ...)
haozhun
force-pushed
the
hive-split
branch
4 times, most recently
from
924793b to
51fc023
Compare
InternalHiveSplit avoids unnecessary duplicate information, and is more friendly to memory accounting. It is used for buffering discovered splits inside Hive connector.
4 participants
No description provided.