Fix an integral float overflow for very large strings #243
Conversation
| val dec = decBuf._1 | ||
| val buf = decBuf._2 | ||
| val maxSpace = (b.length * dec.maxCharsPerByte).toInt + 1 | ||
| val maxSpace = (b.length * (dec.maxCharsPerByte.toDouble)).toInt + 1 |
There was a problem hiding this comment.
I'd put a comment, that for cases where it's possible to have 2B, just try using 2B. I'd also drop the () and maybe just toDouble on the length, since that's the thing that can be large.
There was a problem hiding this comment.
actually, the bug is that dec.maxCharsPerByte returns a Float. That can only store all integers up to 2^{24}. After that it steps up by 2. So the toInt truncates by 2 in that case, not rounding down by one.
We need to convert that Float to a Double before the *.
I don't know what 2B means in this case. Can you explain?
There was a problem hiding this comment.
2B = 2 billion (1 << 31), since that's how toInt handles larger values.
But yeah, Float -> Double makes more sense than integral -> Double, so that seems fine.
Fix an integral float overflow for very large strings
we hit this with a 70MB string. Work around by removing a byte. Better to fix.