What to do with UnbufferedCharStream for 4.7?

[parrt]

@bhamiltoncx Is there a way we can easily update UnbufferedCharStream so that it uses the new 32-bit functionality? it would be okay to get the implementation out and replace with a delegation to the new stuff. there would be no concern about buffer size in memory.

[bhamiltoncx]

I'll take a look!

…

On Wed, Mar 29, 2017, 1:23 PM Terence Parr ***@***.***> wrote: @bhamiltoncx <https://github.com/bhamiltoncx> Is there a way we can easily update UnbufferedCharStream so that it uses the new 32-bit functionality? it would be okay to get the implementation out and replace with a delegation to the new stuff. there would be no concern about buffer size in memory. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1795>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApAU3FSsleCtIymvgvLR9K6D6hw4B-bks5rqqGTgaJpZM4MtXjm> .

[bhamiltoncx]

Just so I understand the architecture of UnbufferedCharStream: it seems like it only ever reads forward, so it can never support seeking backwards. Is that correct? If so, it should be easy enough to port it to use code points.

[bhamiltoncx]

Ah, yeah, it explicitly throws if you do anything but a 1-character lookbehind:

    @Override
    public int LA(int i) {
        if ( i==-1 ) return lastChar; // special case                                                                                       
        sync(i);
        int index = p + i - 1;
        if ( index < 0 ) throw new IndexOutOfBoundsException();

Hmm. So, it would be pretty easy to change char data[] to int data[] and decode UTF-16 surrogate pairs to Unicode code points, but that means 2x the memory usage.

Do I need to optimize that?

[parrt]

haha. was just thinking that. let's just switch to int. Seems like we'd need to decode utf-8 and utf-16 but utf-32 is out unless we add a new ctor right (read() returns 16 bit int).

Next question: do we leave as standalone class with ctor style construction or add to interface? I think it's not used much so we can just upgrade its guts and not complicated main CharStreams interface.

[bhamiltoncx]

I'm fine with just upgrading its guts. I just finished it, PR incoming.

[bhamiltoncx]

Here we go: #1796

I didn't touch the other languages yet.

[bhamiltoncx]

#1798 for C#. We don't need any changes for any other languages (JS is the only other UTF-16 one and it doesn't have an unbuffered input stream).

[parrt]

Actually, it looks like InputStream.read() returns a byte not 16bit as I thought. anyway, how do we know that the incoming InputStream is not already doing some decoding?

[bhamiltoncx]

InputStream.read() is guaranteed to return raw bytes, just like C read(). It does no decoding.

You have to wrap InputStream with InputStreamReader to decode those bytes to Unicode (in Java's case, UTF-16).

[parrt]

ah! ok, that explains it. Is there anything in the new API that needs documentation concerning what kind of streams can be passed in? I guess one of the biggest problems with the prior constructor interface was that it allowed Readers.