zimdump fails on long URLs

[rgaudin]

here's the tail of the output of zimdump dump --dir /data/mqc somezim.zim:

Error writing file to errors dir. /data/mqc/_exceptions/H%2fs0.wp.com%2f_static%2f??-eJylUu1uwyAMfKERr9VaLT+mPQsEj9HwJTCL8vZzEk3NGi2KtD+Iw3fmfABDEl0MhIHAV5FcNTYUGFIXvSjeOhwfUNOV8gQrmXLR3IUxa6kLGBeVdMe4NHBp5Lr4Om8UK1PO9ljghpRk14sZbeg%2fXFMZKsyGKxmhba7NGVS1Tk8eZrnKMo9QaHT4%2fzb0if5Im1m1GkKOsZIw2erDTh5aZEk2mPKHfBXfNAGf+yRp~3
Exception: Error writing file to errors dir. /data/mqc/_exceptions/H%2fs0.wp.com%2f_static%2f??-eJylUu1uwyAMfKERr9VaLT+mPQsEj9HwJTCL8vZzEk3NGi2KtD+Iw3fmfABDEl0MhIHAV5FcNTYUGFIXvSjeOhwfUNOV8gQrmXLR3IUxa6kLGBeVdMe4NHBp5Lr4Om8UK1PO9ljghpRk14sZbeg%2fXFMZKsyGKxmhba7NGVS1Tk8eZrnKMo9QaHT4%2fzb0if5Im1m1GkKOsZIw2erDTh5aZEk2mPKHfBXfNAGf+yRp~3

Touching that file fails as well

touch: /data/mqc/_exceptions/H%2fs0.wp.com%2f_static%2f??-eJylUu1uwyAMfKERr9VaLT+mPQsEj9HwJTCL8vZzEk3NGi2KtD+Iw3fmfABDEl0MhIHAV5FcNTYUGFIXvSjeOhwfUNOV8gQrmXLR3IUxa6kLGBeVdMe4NHBp5Lr4Om8UK1PO9ljghpRk14sZbeg%2fXFMZKsyGKxmhba7NGVS1Tk8eZrnKMo9QaHT4%2fzb0if5Im1m1GkKOsZIw2erDTh5aZEk2mPKHfBXfNAGf+yRp~3: File name too long

Very long URLs seems like a common use case and I believe it calls for a design change in the way those are written to disk.

Might be related to #190

Note: this is zimdump 2.1.0

[kelson42]

I propose to:

• truncate exception files/directory if they hare too long for the filesystem
• Introduce an exceptions.log files in jsons to track them. This file would for each exception give the original path, the exception path, and the reason of the exception (see Machine readable exception log for zimdump (for dumping all content) #108)

[mgautierfr]

truncate exception files/directory if they hare too long for the filesystem

Care must be taken when truncating directories.
We may have 3 entries :

• "long_directory_xxxxxxxxxxyyyyyy/foo.html"
• "long_directory_xxxxxxxxxxyyyyyy/bar.html"
• "long_directory_xxxxxxxxxxzzzzzz/foo.html"

Both first directories must correctly truncated to the same "short name" ("long_directory~1") but the second must be different ("long_directory~2")

[adamlamar]

Is there an example zim file available that has this problem?