Skip to content

Commit 5afce9b

Browse files
ukyoukyo
authored
Merge pull request #4 from STUkh/master
Major update
2 parents 50bc549 + 5d0f00a commit 5afce9b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+6253
-746
lines changed

‎.gitignore

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,13 @@
1+
# Editors
2+
.vscode/
3+
.DS_*
4+
*.sublime*
5+
.idea
6+
7+
# Project
18
node_modules/
29
dev/
10+
test/_dst1.txt
11+
test/_dst2.txt
12+
lz4/
13+
tmp/

‎.gitmodules

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
[submodule "lz4"]
22
path = lz4
3-
url = git@github.com:Cyan4973/lz4.git
3+
url = https://github.com/lz4/lz4.git

‎.npmignore

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
.gitignore
22
dev/
3+
tmp/
34
example/
45
lz4/
56
node_modules/
67
test/
7-
bower.json
8-
Gruntfile.coffee
8+
rollup.config.js

‎Gruntfile.coffee

Lines changed: 0 additions & 99 deletions
This file was deleted.

‎README.md

Lines changed: 146 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
# lz4.js
22

3-
LZ4 for browser
3+
LZ4 WASM, ASM.js and CLI for browser and Node with almost native performance.
4+
This package includes WASM, ASM.js module and variant that can be used in CLI.
45

56
## Install and Use
67

7-
### bower
8-
9-
```
10-
bower install lz4
11-
```
12-
138
html
149

1510
```html
16-
<script src="bower_components/lz4/lz4.js"></script>
11+
<script src="node_modules/lz4/dist/lz4wasm.js"></script>
12+
13+
OR
14+
15+
<script src="node_modules/lz4/dist/lz4asm.js"></script>
16+
1717
```
1818

1919
### npm
@@ -29,95 +29,196 @@ npm install lz4-asm -g
2929
lz4-asm -h
3030
```
3131

32+
### WASM Module modes
33+
34+
### Asynchronouse module mode ( default )
35+
36+
By default module will be exposed as UMD module. In browsers will be defined as `window.lz4init`
37+
Since WASM modules can't be easily loaded synchronously in browsers some setup for initialization have to be proceed:
38+
- `lz4init()` call will trigger asyncronous loading of `.wasm` file and return instance of lz4module
39+
- `lz4module` accessible right after lz4init call, but the only safe method to ensure that `.wasm` file loading finished - wait for lz4module.ready Promise
40+
- `lz4module.ready` Promise will return fully loaded lz4module instance that can be named `lz4`
41+
- From usage perspective not really useful to work directly with `wasm` module, there is predefined `lz4.lz4js` property that contains all useful methods and helpers.
42+
43+
In short all this flow can be described with next code example:
44+
```javascript
45+
const lz4module = lz4init();
46+
lz4module.ready.then((lz4) => {
47+
const lz4js = lz4.lz4js;
48+
})
49+
```
50+
51+
Or even shorter with async/await and Object destructuring
52+
```javascript
53+
const { lz4js } = await lz4init().ready;
54+
```
55+
56+
### Synchronouse module mode
57+
58+
First of all to compile sources in synchronous module mode - set `-s WASM_ASYNC_COMPILATION=0` option in `gulp/emscripten.js` file for `DEV_ARGS` variable.
59+
60+
Then in node env it can be accessed easily `const lz4 = lz4init();` that's it.
61+
BUT! There is a big issue in browsers because WASM modules are restricted for synchronous load in main thread.
62+
There are two ways to load it, both require to pass argument into init function `lz4init({ wasmBinary: wasmModuleAsArrayBuffer });` where wasmModuleAsArrayBuffer - preloaded file as ArrayBuffer in Uint8Array view representation:
63+
1. Manually load/preload module through XHR/fetch, etc. then transform to arrayBuffer and call
64+
```javascript
65+
fetch('PATH_TO_WASM_FILE')
66+
.then(response => response.arrayBuffer())
67+
.then(wasmModuleAsArrayBuffer => lz4init({ wasmBinary: wasmModuleAsArrayBuffer }))
68+
.catch(err => {
69+
throw new Error('Failed to load WASM module');
70+
})
71+
```
72+
1. Another variant - to encode `.wasm` module as base64, include in bundle, on runtime decode base64 to arrayBuffer and init
73+
```javascript
74+
function _base64ToArrayBuffer(base64) {
75+
const binary_string = window.atob(base64);
76+
const binary_length = binary_string.length;
77+
const bytes = new Uint8Array(binary_length);
78+
for (var i = 0; i < binary_length; i++) {
79+
bytes[i] = binary_string.charCodeAt(i);
80+
}
81+
return bytes.buffer;
82+
}
83+
84+
const wasmModuleAsArrayBuffer = _base64ToArrayBuffer(wasmAsBase64);
85+
const lz4 = lz4init({ wasmBinary: wasmModuleAsArrayBuffer });
86+
```
87+
88+
### ASM.js Module modes
89+
90+
Both `async` (default) and `sync` modes available for ASM.js module but with one important difference.
91+
For `sync` variant there **NO NEED for any extra setup**. Set `-s WASM_ASYNC_COMPILATION=0` option in `gulp/emscripten.js` and simply use as:
92+
```javascript
93+
const lz4 = lz4init();
94+
```
95+
96+
It is much easier operate with ASM.js variant is sync mode, but for some cases it operate slower than WASM variant. You can see difference in tests if target different variants in top `require` statement.
97+
3298
## API
3399

34100
### static values
35101

36-
* `lz4.BLOCK_MAX_SIZE_64KB`
37-
* `lz4.BLOCK_MAX_SIZE_256KB`
38-
* `lz4.BLOCK_MAX_SIZE_1MB`
39-
* `lz4.BLOCK_MAX_SIZE_4MB`
102+
- `lz4.BLOCK_MAX_SIZE['64KB']`
103+
- `lz4.BLOCK_MAX_SIZE['256KB']`
104+
- `lz4.BLOCK_MAX_SIZE['1MB']`
105+
- `lz4.BLOCK_MAX_SIZE['4MB']`
40106

41107
### lz4.compress(source, options)
42108

43109
compress to a lz4 buffer.
44110

45-
* source `Uint8Array | Buffer`
46-
* options
47-
* compressionLevel `number` (range of `0-16`, default is `0`)
48-
* blockMaxSize `number` (`lz4.BLOCK_MAX_SIZE_XX`, default is `lz4.BLOCK_MAX_SIZE_4MB`)
49-
* blockIndependent `boolean` (default is false)
50-
* contentChecksum `boolean` (default is false)
51-
* return `Uint8Array | Buffer`
111+
- source `Uint8Array | Buffer`
112+
- options
113+
- frameInfo
114+
- blockSizeID: `Number [4-7]` or as alias `lz4.BLOCK_MAX_SIZE["4MB"]`. Available sizes: `{ "64KB": 4, "256KB": 5, "1MB": 6, "4MB" 7 }`
115+
- blockMode: `Number [0-1]. Default: 0`, 0: blocks linked, 1: blocks independent
116+
- contentChecksumFlag: `Number [0-1]. Default: 0`, 0: Checksum not available. 1: frame terminated with 32-bit checksum of decompressed data
117+
- dictID: `Number [0-1]. Default: 0`, Dictionary ID, sent by compressor to help decoder select correct dictionary; 0: no dictID provided
118+
- blockChecksumFlag: `Number [0-1]. Default: 0`, 1: each block followed by a checksum of block's compressed data; 0: disabled
119+
- preferences
120+
- compressionLevel: `Number [0-16]. Default: 0`, 0: faster but low compression, 16: slower but best compression
121+
- autoFlush: `Number [0-1]. Default: 1`, 1: always flush, reduces usage of internal buffers. 0 - flush disabled
122+
- favorDecSpeed: `Number [0-1]. Default: 1`, 1: parser favors decompression speed vs compression ratio. Only works for high compression modes
123+
- return `Uint8Array | Buffer`
124+
125+
More about options you can find at LZ4 frame description: https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md
52126

53127
### lz4.decompress(source)
54128

55129
decompress a lz4 buffer.
56130

57-
* source `Uint8Array | Buffer`
58-
* return `Uint8Array | Buffer`
131+
- source `Uint8Array | Buffer`
132+
- return `Uint8Array | Buffer`
59133

60134
### lz4.createCompressStream(options)
61135

62136
create a nodejs transform stream.
63137

64-
* options
65-
* compressionLevel `number` (range of `0-16`, default is `0`)
66-
* blockMaxSize `number` (`lz4.BLOCK_MAX_SIZE_XX`, default is `lz4.BLOCK_MAX_SIZE_4MB`)
67-
* blockIndependent `boolean` (default is false)
68-
* contentChecksum `boolean` (default is false)
138+
- options
139+
- frameInfo
140+
- blockSizeID: `Number [4-7]` or as alias `lz4.BLOCK_MAX_SIZE["4MB"]`. Available sizes: `{ "64KB": 4, "256KB": 5, "1MB": 6, "4MB" 7 }`
141+
- blockMode: `Number [0-1]. Default: 0`, 0: blocks linked, 1: blocks independent
142+
- contentChecksumFlag: `Number [0-1]. Default: 0`, 0: Checksum not available. 1: frame terminated with 32-bit checksum of decompressed data
143+
- dictID: `Number [0-1]. Default: 0`, Dictionary ID, sent by compressor to help decoder select correct dictionary; 0: no dictID provided
144+
- blockChecksumFlag: `Number [0-1]. Default: 0`, 1: each block followed by a checksum of block's compressed data; 0: disabled
145+
- preferences
146+
- compressionLevel: `Number [0-16]. Default: 0`, 0: faster but low compression, 16: slower but best compression
147+
- autoFlush: `Number [0-1]. Default: 1`, 1: always flush, reduces usage of internal buffers. 0 - flush disabled
148+
- favorDecSpeed: `Number [0-1]. Default: 1`, 1: parser favors decompression speed vs compression ratio. Only works for high compression modes
149+
150+
More about options you can find at LZ4 frame description: https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md
69151

70152
### lz4.createDecompressStream()
71153

72154
create a nodejs transform stream.
73155

74-
75156
## Development
76157

77158
### Require
78159

79-
* latest emscripten
80-
* nodejs v5.0~
160+
- latest emscripten
161+
- nodejs 10+
81162

82163
### Init
83164

84-
Get repos.
85-
86-
```
87-
git clone [email protected]:ukyo/lz4.js.git
88-
```
89-
90-
If you don't install the grunt-cli run below.
165+
Clone the repo.
91166

92167
```
93-
npm install -g grunt-cli
168+
git clone https://github.com/ukyo/lz4.js.git
94169
```
95170

96-
Install dev dependencies
171+
Install the dev dependencies.
97172

98173
```
99174
cd path/to/lz4.js
100175
npm install
101176
```
102177

103-
Download original LZ4 repos and compile for development.
178+
Download the original LZ4 repo and compile for development.
104179

105180
```
106-
grunt init
181+
npx gulp init
107182
```
108183

109-
### Write codes
184+
### Write code
110185

111-
Watch codes update and test.
186+
Watch for code updates and run tests.
112187

113188
```
114-
grunt watch
189+
npx gulp watchDev
115190
```
116191

192+
Run development tests
193+
117194
### Release
118195

119196
Release build.
120197

121198
```
122-
grunt release
123-
```
199+
npx gulp release
200+
```
201+
202+
### List of available gulp commands
203+
```
204+
buildLib
205+
cleanTest
206+
cleanRelease
207+
compileDev
208+
compileAsmRelease
209+
compileWasmRelease
210+
concatDev
211+
concatAsmRelease
212+
concatWasmRelease
213+
debugTests
214+
fetchLib
215+
rollup
216+
runTests
217+
watchDev
218+
219+
initTask
220+
release
221+
releaseAsm
222+
releaseWasm
223+
testDev
224+
```

0 commit comments

Comments
 (0)