[124] TiltLoader breaks in a module env

[drcmda]

Repro: https://codesandbox.io/s/elegant-villani-9d3hr?file=/src/index.js

import { TiltLoader } from "three/examples/jsm/loaders/TiltLoader"

new TiltLoader().load("/model.tilt", (scene) => console.log(scene))

This is the error message in a local env (Parcel):

TiltLoader.js:42 TypeError: _jszipModuleMin.JSZip is not a constructor
    at TiltLoader.parse (TiltLoader.js:59)
    at Object.onLoad (TiltLoader.js:32)
    at XMLHttpRequest.<anonymous> (three.module.js:36749)

[drcmda]

PS. it looks like the import statement is wrong, TiltLoader imports jszip as:

import { JSZip } from 'three/examples/jsm/libs/jszip.module.min.js'

but the library doesn't seem to have a named export called "JSZip", it's undefined. The default export on the other hand yields a function:

import JSZip from 'three/examples/jsm/libs/jszip.module.min.js'

Btw, I generally wonder why these aren't npm dependencies. jszip is here: https://www.npmjs.com/package/jszip Why would threejs use a local copy? If my app already happens to use jszip, it would now serve it twice, not to mention the headache on your side keeping these dependencies up to date.

[Mugen87]

import JSZip from 'three/examples/jsm/libs/jszip.module.min.js'

If we do this, the example would report:

Uncaught SyntaxError: The requested module '../libs/jszip.module.min.js' does not provide an export named 'default'

So it seems the current import is correct. Could this be a problem of Parcel?

Why would threejs use a local copy?

We do not want to force users into using npm for just working with the example code. Ideally, they can download the repository and host it on a local webserver.

[drcmda]

Could this be a problem of Parcel?

I looked, it's a problem with JSzip: Stuk/jszip#349 The lib appears unmaintained and i don't think they will fix it. In that case the loader won't be of much use imo.

Some people have suggested to use https://github.com/101arrowz/fflate instead.

Other projects have moved on as well for the same reason, this PR explains some of the problems with JSZip:

• require cycle
• missing support for es6 modules
• incompatible with Rollup (Parcel, CSB, ...)

We do not want to force users into using npm for just working with the example code. Ideally, they can download the repository and host it on a local webserver.

Not complaining, i was just curious, i just hope at some point in the future this will be re-considered because JSM has incredible value that is held back atm.

[drcmda]

Just found this: https://www.npmjs.com/package/@progress/jszip-esm

This is a fork maintained by Telerik where they have fixed it. If you use their package, that's the easiest solution.

[Mugen87]

We could not upgrade JSZip or use forks so far since we rely on an older version of the lib that supports sync unzip. We can only switch to a new library if it also supports this feature.

[drcmda]

Understood, wasn't aware they changed the api that drastically. Fflate supports both async and sync, for instance fflate.unzipSync(...). It's smaller (~5kb vs 80kb), faster (they say up to 50%), no module bugs, maintained, seems like a good replacement. For me it's not so pressing, just wanted to try out TiltLoader really, but maybe if time allows it could be amended. =)

[mrdoob]

Switching to fflate sounds good to me! 👍

[101arrowz]

Hi, author of fflate here! The reason the async version exists in JSZip is beyond me, since it doesn't actually offload the processing to other threads. On the other hand, if the assets in the ZIP are reasonably large (around 500kB), fflate will use worker threads to compress/decompress them faster.

However, I looked at some of the demos using JSZip and they don't seem to load that much data, so using the synchronous unzipping algorithm would be no different to the async version. You could even replace gunzip.js, which is used by the NRRD loader, with fflate because gunzip and unzip use the same core algorithm.

Since it seems the ThreeJS team wants minified ES6 module scripts, I've hand-tuned a minified version of the library and pasted it below. It's 4.4kB (2.2kB gzipped) and supports synchronous ZIP decompression (through the exported unzipSync method) and GZIP decompression (through gunzipSync).

Usage:

import { unzipSync, strFromU8 } from '../libs/fflate.module.min.js';
let arrayBuffer = ... // From XHR, presumably
const zip = unzipSync(new Uint8Array(arrayBuffer));

// If we were using JSZip, the following would be:
// zip.files['data.xml'].asUint8Array()
const dataXmlU8 = zip['data.xml'];
// zip.files['data.xml'].asArrayBuffer()
const dataXmlBuffer = dataXmlU8.buffer;
// zip.files['data.xml'].asText()
const dataXmlText = strFromU8(dataXmlU8);

import { gunzipSync } from '../libs/fflate.module.min.js';
let arrayBuffer = ... // From XHR, presumably

// If we were using gunzip.js, the following would be:
// const gunzip = new Zlib.Gunzip(new Uint8Array(arrayBuffer));
// const data = gunzip.decompress();
const data = gunzipSync(new Uint8Array(arrayBuffer));

Minified code (to put in fflate.module.min.js):

/*!
fflate - fast JavaScript compression/decompression
<https://101arrowz.github.io/fflate>
Licensed under MIT. https://github.com/101arrowz/fflate/blob/master/LICENSE
Subset included: synchronous unzip and gunzip
*/
"use strict";var r=Uint8Array,n=Uint16Array,e=Uint32Array,t=new r([0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,0,0,0]),a=new r([0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,0,0]),i=new r([16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15]),f=function(r,e){for(var t=new n(31),a=0;a<31;++a)t[a]=e+=1<<r[a-1];return t},o=f(t,2);o[28]=258;for(var u=f(a,0),v=new n(32768),l=0;l<32768;++l){var c=(43690&l)>>>1|(21845&l)<<1;c=(61680&(c=(52428&c)>>>2|(13107&c)<<2))>>>4|(3855&c)<<4,v[l]=((65280&c)>>>8|(255&c)<<8)>>>1}var w=function(r,e){for(var t=r.length,a=0,i=new n(e);a<t;++a)++i[r[a]-1];var f=new n(1<<e),o=new n(e);for(a=0;a<e;++a)o[a]=o[a-1]+i[a-1]<<1;var u=15-e;for(a=0;a<t;++a)if(r[a])for(var l=a<<4|r[a],c=e-r[a],w=o[r[a]-1]++<<c,h=w|(1<<c)-1;w<=h;++w)f[v[w]>>>u]=l;return f},h=new r(288);for(l=0;l<144;++l)h[l]=8;for(l=144;l<256;++l)h[l]=9;for(l=256;l<280;++l)h[l]=7;for(l=280;l<288;++l)h[l]=8;var d=new r(32);for(l=0;l<32;++l)d[l]=5;var s=w(h,9),g=w(d,5),p=function(r){for(var n=r[0],e=1;e<r.length;++e)r[e]>n&&(n=r[e]);return n},y=function(r,n,e){var t=n/8>>0;return(r[t]|r[t+1]<<8)>>>(7&n)&e},b=function(r,n){var e=n/8>>0;return(r[e]|r[e+1]<<8|r[e+2]<<16)>>>(7&n)},x=function(r){return(r/8>>0)+(7&r&&1)},m=function(t,a,i){(null==a||a<0)&&(a=0),(null==i||i>t.length)&&(i=t.length);var f=new(t instanceof n?n:t instanceof e?e:r)(i-a);return f.set(t.subarray(a,i)),f},C=function(n,e,f){var v=n.length,l=!e||f,c=!f||f.i;f||(f={}),e||(e=new r(3*v));var h=function(n){var t=e.length;if(n>t){var a=new r(Math.max(2*t,n));a.set(e),e=a}},d=f.f||0,C=f.p||0,S=f.b||0,k=f.l,F=f.d,z=f.m,E=f.n;if(d&&!k)return e;var O=8*v;do{if(!k){f.f=d=y(n,C,1);var U=y(n,C+1,3);if(C+=3,!U){var A=n[(K=x(C)+4)-4]|n[K-3]<<8,D=K+A;if(D>v){if(c)throw"unexpected EOF";break}l&&h(S+A),e.set(n.subarray(K,D),S),f.b=S+=A,f.p=C=8*D;continue}if(1==U)k=s,F=g,z=9,E=5;else{if(2!=U)throw"invalid block type";var T=y(n,C,31)+257,M=y(n,C+10,15)+4,j=T+y(n,C+5,31)+1;C+=14;for(var q=new r(j),B=new r(19),G=0;G<M;++G)B[i[G]]=y(n,C+3*G,7);C+=3*M;var H=p(B),I=(1<<H)-1;if(!c&&C+j*(H+7)>O)break;var J=w(B,H);for(G=0;G<j;){var K,L=J[y(n,C,I)];if(C+=15&L,(K=L>>>4)<16)q[G++]=K;else{var N=0,P=0;for(16==K?(P=3+y(n,C,3),C+=2,N=q[G-1]):17==K?(P=3+y(n,C,7),C+=3):18==K&&(P=11+y(n,C,127),C+=7);P--;)q[G++]=N}}var Q=q.subarray(0,T),R=q.subarray(T);z=p(Q),E=p(R),k=w(Q,z),F=w(R,E)}if(C>O)throw"unexpected EOF"}l&&h(S+131072);for(var V=(1<<z)-1,W=(1<<E)-1,X=z+E+18;c||C+X<O;){var Y=(N=k[b(n,C)&V])>>>4;if((C+=15&N)>O)throw"unexpected EOF";if(!N)throw"invalid length/literal";if(Y<256)e[S++]=Y;else{if(256==Y){k=null;break}var Z=Y-254;if(Y>264){var $=t[G=Y-257];Z=y(n,C,(1<<$)-1)+o[G],C+=$}var _=F[b(n,C)&W],rr=_>>>4;if(!_)throw"invalid distance";if(C+=15&_,R=u[rr],rr>3&&($=a[rr],R+=b(n,C)&(1<<$)-1,C+=$),C>O)throw"unexpected EOF";l&&h(S+131072);for(var nr=S+Z;S<nr;S+=4)e[S]=e[S-R],e[S+1]=e[S+1-R],e[S+2]=e[S+2-R],e[S+3]=e[S+3-R];S=nr}}f.l=k,f.p=C,f.b=S,k&&(d=1,f.m=z,f.d=F,f.n=E)}while(!d);return S==e.length?e:m(e,0,S)},S=function(r,n){return r[n]|r[n+1]<<8},k=function(r,n){return(r[n]|r[n+1]<<8|r[n+2]<<16)+2*(r[n+3]<<23)},F=function(r){var n="";if("undefined"!=typeof TextDecoder)return(new TextDecoder).decode(r);for(var e=0;e<r.length;){var t=r[e++];t<128?n+=String.fromCharCode(t):t<224?n+=String.fromCharCode((31&t)<<6|63&r[e++]):t<240?n+=String.fromCharCode((15&t)<<12|(63&r[e++])<<6|63&r[e++]):(t=((15&t)<<18|(63&r[e++])<<12|(63&r[e++])<<6|63&r[e++])-65536,n+=String.fromCharCode(55296|t>>10,56320|1023&t))}return n};export var strFromU8=F;var z=function(r,n){S(r,n+8);var e=S(r,n+10),t=k(r,n+20),a=k(r,n+24),i=S(r,n+28);return[t,e,a,F(r.subarray(n+46,n+46+i)),n+46+i+S(r,n+30)+S(r,n+32),k(r,n+42)]};export var unzipSync=function(n){for(var e={},t=n.length-22;101010256!=k(n,t);--t)if(!t||n.length-t>65558)throw"invalid zip file";var a=S(n,t+8);if(!a)return{};for(var i=k(n,t+16),f=0;f<a;++f){var o=z(n,i),u=o[0],v=o[1],l=o[2],c=o[3],w=o[4],h=o[5],d=h+30+S(n,h+26)+S(n,h+28);if(i=w,v){if(8!=v)throw"unknown compression type "+v;e[c]=C(n.subarray(d,d+u),new r(l))}else e[c]=m(n,d,d+u)}return e};export var gunzipSync=function(n){return C(n.subarray(function(r){if(31!=r[0]||139!=r[1]||8!=r[2])throw"invalid gzip data";var n=r[3],e=10;4&n&&(e+=r[10]|2+(r[11]<<8));for(var t=(n>>3&1)+(n>>4&1);t>0;t-=!r[e++]);return e+(2&n)}(n),-8),new r(function(r){var n=r.length;return(r[n-4]|r[n-3]<<8|r[n-2]<<16)+2*(r[n-1]<<23)}(n)))}

Btw, I generally wonder why these aren't npm dependencies. jszip is here: https://www.npmjs.com/package/jszip Why would threejs use a local copy? If my app already happens to use jszip, it would now serve it twice, not to mention the headache on your side keeping these dependencies up to date.

This duplication issue will still exist if the examples were to use fflate, but since most users seem to think of 4kB as pocket change, it probably doesn't matter.

[Mugen87]

The library works with TiltLoader but using it with 3MFLoader produces a runtime error:

invalid zip file

This is the place where it happens:

three.js/examples/jsm/loaders/3MFLoader.js

Line 115 in 6a05d60

zip = new JSZip( data ); // eslint-disable-line no-undef

Here is the current example of 3MFLoader: https://threejs.org/examples/webgl_loader_3mf

[Mugen87]

And here is the 3MF example on my dev branch using fflate:

https://raw.githack.com/Mugen87/three.js/dev1/examples/webgl_loader_3mf.html

[101arrowz]

The issue there is that in your source code, you pass an ArrayBuffer to fflate.unzipSync but you need to pass a Uint8Array. To fix the problem, use fflate.unzipSync(new Uint8Array(data)). I applied this hotfix within DevTools and the assets were displayed fine after that. Don't worry about performance cost, this is a free operation (just creates an interface to view the same chunk of memory without any copying).

[Mugen87]

Okay, webgl_loader_3mf works now but I get an empty object zip object in webgl_loader_3mf_materials. Please check out:

https://raw.githack.com/Mugen87/three.js/0b388b5f6fa9228425acae87775d3a28088501a9/examples/webgl_loader_3mf_materials.html

Current prod version: https://threejs.org/examples/webgl_loader_3mf_materials

[101arrowz]

That's a bug in either the compressed version or the actual library. Looking into it now. Thanks for the help with debugging.

EDIT: This is actually the result of that file being Zip64. fflate has no plans to support this format because it's designed for files above 4 GB compressed, and JavaScript typed arrays cannot be that big. This can be resolved by unzipping and rezipping the file; when this is done, the unnecessary Zip64 data is removed and a standard ZIP file is produced, which can be read by fflate. @Mugen87 is this a feasible compromise? If not I can probably hack together a Zip64 reader in a few days.

[Mugen87]

@Mugen87 is this a feasible compromise?

I'm afraid no since we can't ask users to unpack and pack 3MF files again. Such things have to work out of the box.

Besides, if JSZip supports Zip64 I expect to have proper supported in other zip libs, too.