New formats support

[aonez]

Some compression formats that could be added. Being in the list does not mean they will be added, just taked into account:

• wim -> 1.2.0r 3840+
• lha
• lzh
• lz4 Add lz4 support #85
• lz5 https://github.com/inikep/lizard [REQ] LZ5 / Lizard format support #1127
• lzo lzo support #6
• lrzip (asked here) [REQ] Add support for lrz / lrztar #549 -> 1.2.0r 3806+
• brotli (asked here): beta
• cpio and cpgz Issues with uncompressing .cpgz #151 (probably using bsdtar)
• zstandard: beta
• cso Maybe add support for .cso (compressed ISO) in some future version #154
• uu (https://en.wikipedia.org/wiki/Uuencoding)
• zpaq (LRZIP as suggested by @MaxPower85) -> 1.2.0r 3806+ LRZIP in slow method
• compressed ISO (asked by Andy via mail), similar as CDImage
• ace extraction support ACE archive support #402 -> 1.2.0

Just extraction:

• sit (Stuffit) Opening .sit (StuffIt) files? #240
• text-to-binary Additional Formats #338

[dezzeus]

It would be nice to also have Zstandard.

[MaxPower85]

Since you have lrzip on the list, add zpaq too since lrzip can optionally use zpaq for it's 2nd stage... but you can use zpaq independently too.

You can also add rzip... lrzip is similar, but it's not the same format.

Maybe add Apple's lzfse too... but I'm not sure did Apple mean it to be used on it's own as a format for archives (or did they mean it to just be used within some other formats), since I can't find info about what kind of extension could be used for archives compressed with lzfse... although you can compress some file or a tar archive with lzfse and it seems pretty good for a format that doesn't use multithreading... and people are saying that lzfse is supposed to be energy efficient... but even the file command on Sierra doesn't seem to recognize what type of archive is that if you compress files with lzfse...

https://github.com/lzfse/lzfse

If you look at a file compressed with lzfse in some HexEditor, it says "bvx2" at the begining... and here's a clue about what that means: https://github.com/lzfse/lzfse/blob/497c5c176732769abf36ccc71a31c06bad93a84d/src/lzfse_internal.h#L276-L281

So it doesn't seem that it would be difficult to recognize lzfse compressed archives... but the question is did Apple intend for it to be used just on it's own like bzip2 or gzip.

It can also be used for compressed .dmg images when you create a compressed .dmg with hdiutil and you use -format ULFO like hdiutil create -volname vol_name -srcfolder source_folder -ov -format ULFO new_dmg_image.dmg

I'm reading that 7z beta for Windows has added support for .dmg images that use lzfse compression... but 7z for macOS or Linux doesn't seem to recognize them yet.

[yetisyny]

The .WIM format (Windows Imaging Format) has been supported for both compression and decompression by 7-Zip for Windows for several years. Since it is part of the relatively short list of filetypes 7-Zip for Windows supports not only reading but also writing to, even in the GUI, it ought to be included in Keka for feature parity with the Windows version of 7-Zip. There is also already a library and utility for the .WIM format that is cross-platform, at https://wimlib.net/, although this library is under GNU GPL version 3 so you cannot use it legally unless you start using that license too which I doubt you would want to do.

So using the 7-Zip implementation would probably work better license-wise. And actually the 7-Zip implementation for the .WIM format is already included in the p7zip ports to UNIX-based operating systems (including macOS, Linux, etc.). So directly using p7zip is probably the easiest way to do this, in fact you already use p7zip for other things. And as far as the virtues of the .WIM format or why anyone would want to use it, it is a file-based imaging format that can archive advanced filesystem features and can be used with several different compression algorithms, and is in widespread use, especially by Microsoft which uses it for everything. Plus it is the ONLY compression format supported by the GUI and command-line versions 7-Zip for Windows which Keka does not also support, so adding it would bring Keka to feature parity with 7-Zip regarding supported formats to compress to, and of course it is also there in p7zip too. The other formats 7-Zip advertises on its website as being able to compress besides .WIM are 7Z, XZ, BZIP2, GZIP, TAR, ZIP, and you already support all of those! (I think 7-Zip also supports maybe a few more such as ISO but those are not mentioned there, anyway you already support ISO too.)

[aonez]

this library is under GNU GPL version 3 so you cannot use it legally

If that is true, then lbzip2 also can't be bundled within Keka.

[d235j]

Regarding bundling, please see https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation. If the proprietary components are not linking to the GPL components, then you should be OK; however, you need to provide source code to the GPL components.

[aonez]

Thanks @d235j, you're right. Already started pushing the GPL code here 😊

[magitk]

+1 for zpaq

[dh1337]

any news on brotli?

[gingerbeardman]

@denishamann1337 out of interest what is your use case for Brotli? What would it achieve that other existing formats or schemes can not?

[p2k]

+1 for zpaq

It is the best pack format I know combining deduplication and a strong compression that outperforms every competitor. It actually allows multiple versions of the same file(s) so it is suitable for incremental backups. Needless to say it offers industrial standard encryption.

Having a GUI for zpaq would be a bliss, but is considerably harder to do than for all the other formats since it has some unique features (like the aforementioned multi-version capability).

More information on zpaq: http://mattmahoney.net/dc/zpaq.html