noVnc does not handle authentication correctly when rfb 3.3 is used

[Cedware]

When running a vnc server that uses the RFB Version 3.3 and authentication is disabled everything is fine. When the password authentication is enabled, noVnc does shows the password dialog, but after that noVnc seems to handle the Protocol wrong. After entering the password the remote display is not shown and in the browser log you can see that noVnc prints completely wrong screen information and then detects an unsupported encoding:

Data URI scheme cursor supported [browser.js:37:13](http://localhost:8080/core/util/browser.js)
>> RFB.constructor [rfb.js:210:13](http://localhost:8080/core/rfb.js)
>> Display.constructor [display.js:26:13](http://localhost:8080/core/display.js)
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0 [display.js:56:13](http://localhost:8080/core/display.js)
<< Display.constructor [display.js:58:13](http://localhost:8080/core/display.js)
New state 'connecting', was ''. [rfb.js:889:13](http://localhost:8080/core/rfb.js)
>> RFB.connect [rfb.js:546:13](http://localhost:8080/core/rfb.js)
connecting to ws://localhost:5901/websockify [rfb.js:549:17](http://localhost:8080/core/rfb.js)
<< RFB.connect [rfb.js:600:13](http://localhost:8080/core/rfb.js)
<< RFB.constructor [rfb.js:280:13](http://localhost:8080/core/rfb.js)
GET
http://localhost:8080/favicon.ico
[HTTP/1.1 404 Not Found 1ms]

>> WebSock.onopen [websock.js:251:17](http://localhost:8080/core/websock.js)
Starting VNC handshake [rfb.js:640:17](http://localhost:8080/core/rfb.js)
<< WebSock.onopen [websock.js:257:17](http://localhost:8080/core/websock.js)
Server ProtocolVersion: 003.003 [rfb.js:1356:13](http://localhost:8080/core/rfb.js)
Sent ProtocolVersion: 003.003 [rfb.js:1396:13](http://localhost:8080/core/rfb.js)
Authenticating using scheme: 2 [rfb.js:1462:13](http://localhost:8080/core/rfb.js)
Screen: 0x0, bpp: 10, depth: 0, bigEndian: 5, trueColor: 160, redMax: 8216, greenMax: 255, blueMax: 255, redShift: 0, greenShift: 255, blueShift: 0 [rfb.js:2050:13](http://localhost:8080/core/rfb.js)
Setting viewport to full display region [display.js:144:17](http://localhost:8080/core/display.js)
New state 'connected', was 'connecting'. [rfb.js:889:13](http://localhost:8080/core/rfb.js)
Failed while connected: Unsupported encoding (encoding: 828452912) [rfb.js:937:21](http://localhost:8080/core/rfb.js)
New state 'disconnecting', was 'connected'. [rfb.js:889:13](http://localhost:8080/core/rfb.js)
>> RFB.disconnect [rfb.js:604:13](http://localhost:8080/core/rfb.js)
>> Keyboard.allKeysUp [keyboard.js:250:13](http://localhost:8080/core/input/keyboard.js)
<< Keyboard.allKeysUp [keyboard.js:254:13](http://localhost:8080/core/input/keyboard.js)
Closing WebSocket connection [websock.js:277:21](http://localhost:8080/core/websock.js)
<< RFB.disconnect [rfb.js:633:13](http://localhost:8080/core/rfb.js)
New state 'disconnected', was 'disconnecting'. [rfb.js:889:13](http://localhost:8080/core/rfb.js)
Clearing disconnect timer [rfb.js:892:17](http://localhost:8080/core/rfb.js)
>> WebSock.onclose [websock.js:261:17](http://localhost:8080/core/websock.js)
<< WebSock.onclose [websock.js:263:17](http://localhost:8080/core/websock.js)

To Reproduce
Steps to reproduce the behavior:

1. Install x11vnc and websockify
2. Run x11vnc -storepasswd to create a password for the vnc server
3. Launch x11vnc x11vnc -display :0 -rfbport 5902 -rfbversion 3.3 -rfbauth ~/.vnc/passwd, make sure to adopt the "-display parameter for your setup
4. Start websockify websockify 5901 127.0.0.1:5902
5. Modify the vnc_lite.html file to use the url 'ws://localhost:5901/websockify'
6. Browse to the vnc_lite.html and enter the password created in step 2
7. You should experience the behavior described above

Expected behavior
I expect to see the remote display after entering the correct password.

Client (please complete the following information):

• OS: Manjaro Linux
• Browser: Firefox
• Browser version: 112.0

Server (please complete the following information):

• noVNC version: v1.4.0
• VNC server: x11vnc 0.9.16
• WebSocket proxy: websockify(python version) 0.11.0

Additional context
Initialy this problem appeared with a different, proprietary, vnc server, so x11vnc shouldn't be the issue.
The Problem seems to be cause by this code segment: https://github.com/novnc/noVNC/blob/v1.4.0/core/rfb.js#L2021
When I remove the if-condition with its body, noVnc behaves correctly for my use case, but I'm not aware of the implications of this change,

Edit: Provided correct noVNC version and provided a link to the code segment, causing the problem.

[CendioOssman]

Yeah, I think that line might be misplaced. I think it's the security reason, not the security result, that is missing in those older versions.

Should probably double-check the RealVNC implementations, as those are the reference.

[CendioOssman]

This was apparently a regression in 1.4.0 caused by 5671072. It's been fixed in 72f6810 and 370f21b. Tested against TigerVNC this time, so it should hopefully be more correct.