Best strategy to "save as"?

[aral-matrix]

I am honestly a bit lost in the libzip documentation, most of the functions I have to learn by trial & error / copying code from examples. So my apologies if I am missing something obvious, but:

How would I go about modifying an existing zip file and then saving the modified archive to a NEW file?

Currently, my understanding is that

1. libzip modifies an archive in-place, meaning it provides file descriptors for which the data is read & decompressed from the source file on-demand ("just in time")
2. modified files / added file contents are stored in memory (until zip_close is called)
3. zip_close - if successful - will automatically store modifications to the archive in the source file

Accordingly, it is not possible to read changes that have been written to the archive (zip_t) without first saving and re-opening (not so bad, just prevents direct plausibility checks after a write).

Furthermore, it appears that - in order to accomplish a "save as" function, I would have to create a new archive from scratch, and add all the files to that new archive for saving.

Is that really how I need to do it? So - assuming I want to provide a User API "save as" function, and the user calls that repeatedly for different file names - does that also mean I have to re-create the full archive each time a new save file is desired?

Or is
https://github.com/aseprite/libzip/blob/master/examples/in-memory.c
the way to do it, i.e., I need to work with the archive raw data to save it?

Edit: I just managed to "save as" with the in-memory example, but that raises the next question: How do I prevent the zip_close? Or is re-opening via zip_open_from_source a non-issue in terms of performance, even for large archives?

[aral-matrix]

Here is my implementation for OpenXLSX, I hope I don't get shunned for doing a CPP wrapper:
https://github.com/troldal/OpenXLSX/blob/ktb_zippy/Tests/ziptest.cpp

It appears that it is unavoidable to wrap a save operation in the below calls, as done starting from
https://github.com/troldal/OpenXLSX/blob/ktb_zippy/Tests/ziptest.cpp#L349 (Edit: function ZipArchive::save, in case the next commit changes the line again)

1. ensure that zip src remains usable after zip_close, by increasing the refcount: zip_source_keep
2. commit modifications by closing the zip archive: zip_close
3. perform the actual read-out of archive raw data:

• determine the raw data size of the (modified) archive zip_source_stat
• open the src for raw data read: zip_source_open
• read out the raw data: zip_source_read
• close the src: zip_source_close

4. re-open the zip archive to be ready for further read access and/or modifications: zip_open_from_source

[dillof]

I am honestly a bit lost in the libzip documentation, most of the functions I have to learn by trial & error / copying code from examples. So my apologies if I am missing something obvious, but:

I'm sad to hear that. If you can give us more concrete feedback, we can work on improving it.

How would I go about modifying an existing zip file and then saving the modified archive to a NEW file?

This is currently not supported directly, but wouldn't be hard to add. We'll discuss this internally and get back to you.

Currently, my understanding is that

1. libzip modifies an archive in-place, meaning it provides file descriptors for which the data is read & decompressed from the source file on-demand ("just in time")

in-place might be misleading here: libzip writes the new archive to a temporary file and once complete, replaces the original file with that.

2. modified files / added file contents are stored in memory (until zip_close is called)

No, they are only read from the supplied source when zip_close() is called.

3. zip_close - if successful - will automatically store modifications to the archive in the source file

True, see above for details.

Accordingly, it is not possible to read changes that have been written to the archive (zip_t) without first saving and re-opening (not so bad, just prevents direct plausibility checks after a write).

You can read changed data by calling zip_fopen() and not supplying the flag ZIP_FL_UNCHANGED.

Furthermore, it appears that - in order to accomplish a "save as" function, I would have to create a new archive from scratch, and add all the files to that new archive for saving.

Currently, yes. Unless you're using a filesystem that supports file cloning (btrfs on Linux and APFS on macOS), this is basically what zip_close() does anyway. (On file systems that support cloning, the file is only rewritten starting with the first changed entry.)

Is that really how I need to do it? So - assuming I want to provide a User API "save as" function, and the user calls that repeatedly for different file names - does that also mean I have to re-create the full archive each time a new save file is desired?

Yes. However, unless you're using a file system that supports cloning, a new file will have to be written from scratch for each save anyway.

Or is https://github.com/aseprite/libzip/blob/master/examples/in-memory.c the way to do it, i.e., I need to work with the archive raw data to save it?

Edit: I just managed to "save as" with the in-memory example, but that raises the next question: How do I prevent the zip_close? Or is re-opening via zip_open_from_source a non-issue in terms of performance, even for large archives?

You only have to call zip_close() when you want to write the archive to disk. Like I said above, you can read from the modified archive just fine.

For an archive with many entries, this introduces some overhead.

We've discussed implementing a way to flush the changes to disk (or memory in your case) while keeping the archive open, but that introduces a lot of additional complexity in an already complex section of the code, with many corner cases that could introduce subtle bugs, so we've decided against adding it.

[aral-matrix]

I am honestly a bit lost in the libzip documentation, most of the functions I have to learn by trial & error / copying code from examples. So my apologies if I am missing something obvious, but:

I'm sad to hear that. If you can give us more concrete feedback, we can work on improving it.

Hi & thanks for the timely response!

My feedback is very generic, because I was not able to find documentation that is organized in a way that it reflects the architectural design of libzip.

Basically, I had to download your repository and manually search for functions in the code / read the code to see what function calls which in order to understand when I have to call which function to free / prevent a double free, and to get what data.

What I would think could help users a lot is an introduction page that explains the functions and data structures exposed via the API, their purpose and how they interact.

E.g. it is not very easy to wrap my head around the difference between a zip source and a zip archive - especially with the difference in lifetime (being able to close and reopen an archive while keeping the source alive).

With my limited understanding after two days of using libzip, I would document it like this, to give you an idea, using the example of zip source and zip archive (Disclaimer: my explanation will be wrong due to my limited understanding, but I hope you get the picture):
"A zip source (zip_source_t) provides a structure managing links to all data contained in a zip archive. A zip archive (zip_t) manages the zip archive metadata and functions handling access to archive contents on filename / index basis. A zip archive always requires a valid zip source, whereas a zip source can be used independent of an archive."

Another thing that I am missing is Examples on how and when to use certain functions, and how to assert validity of operations (e.g. I noticed that when a file entry is deleted / marked for deletion, the index remains and the entry name for that index is still accessible, but zip_name_locate fails (correctly so) to find a deleted entry. This is kind of confusing.

If you are considering to improve the overall documentation, I will try to give more feedback whenever something comes to mind.

2. modified files / added file contents are stored in memory (until zip_close is called)

No, they are only read from the supplied source when zip_close() is called.

Ah - okay - I stumbled across that just the past hour when I realized I had created undefined behavior by calling zip_source_buffer with temporary data (string literal) - until I saw in documentation that "[..] data must remain valid for the lifetime of the created source".

Accordingly, it is not possible to read changes that have been written to the archive (zip_t) without first saving and re-opening (not so bad, just prevents direct plausibility checks after a write).

You can read changed data by calling zip_fopen() and not supplying the flag ZIP_FL_UNCHANGED.

I am calling zip_fopen_index for getting entry data, but I am already providing 0 for flags - am I doing something wrong here?
https://github.com/troldal/OpenXLSX/blob/ktb_zippy/Tests/ziptest.cpp#L462

Furthermore, it appears that - in order to accomplish a "save as" function, I would have to create a new archive from scratch, and add all the files to that new archive for saving.

Currently, yes. Unless you're using a filesystem that supports file cloning (btrfs on Linux and APFS on macOS), this is basically what zip_close() does anyway. (On file systems that support cloning, the file is only rewritten starting with the first changed entry.)

As you can see in the above links and my previous comment, I have improved my implementation a little bit, and am now able to accomplish the "save as" with the above logic - zip_source_keep, then closing the zip archive, reading the raw data from the zip source, and re-opening the zip file (function ZipArchive::Save).

My implementation is only using the in-memory manipulation of archives, and - since it is tailored for XLSX - currently only supports file data as std::strings (can be unicode).

For an archive with many entries, this introduces some overhead.

We've discussed implementing a way to flush the changes to disk (or memory in your case) while keeping the archive open, but that introduces a lot of additional complexity in an already complex section of the code, with many corner cases that could introduce subtle bugs, so we've decided against adding it.

Understood :)

[dillof]

I am honestly a bit lost in the libzip documentation, most of the functions I have to learn by trial & error / copying code from examples. So my apologies if I am missing something obvious, but:

I'm sad to hear that. If you can give us more concrete feedback, we can work on improving it.

Hi & thanks for the timely response!

My feedback is very generic, because I was not able to find documentation that is organized in a way that it reflects the architectural design of libzip.

Basically, I had to download your repository and manually search for functions in the code / read the code to see what function calls which in order to understand when I have to call which function to free / prevent a double free, and to get what data.

What I would think could help users a lot is an introduction page that explains the functions and data structures exposed via the API, their purpose and how they interact.

E.g. it is not very easy to wrap my head around the difference between a zip source and a zip archive - especially with the difference in lifetime (being able to close and reopen an archive while keeping the source alive).

We've added a section to libzip(3) describing the central data structures and how they are created/destroyed and when data for added files is accessed (during zip_close()).

Would this have helped you get started, and what else would you like to see added?

[aral-matrix]

We've added a section to libzip(3) describing the central data structures and how they are created/destroyed and when data for added files is accessed (during zip_close()).

Would this have helped you get started, and what else would you like to see added?

Thank you - yes, those short explanations already help to clarify, although I am still a bit fuzzy on zip_source_t, specifically because this description now sounds as if an archive can combine different zip sources (for different input files), without an explicit mention of that. Something like "It is possible to link files in the archive to different sources, from where the data will be accessed when the archive is compressed."

Regarding what else I would like to see - the problem with learning a new interface is you can only do it from scratch once, that moment has passed for me, now I am already somewhat familiar with it and don't remember all of the questions I may have initially had.

I suppose from a developer's perspective, what would have saved me some digging and would still help is an overview over the layout of a zip file and what properties can be accessed (modified) by libzip using which functions.

The below is just a poor example, I hope it is structured well enough that you get the picture :)

--- zip general header ---
property1 -> getProperty1, setProperty1
property2 -> getProperty2, setProperty2
[..]
entryCount -> zip_add, zip_remove
entries [repeats entryCount times]
entryName -> zip_name_locate, ...
offsetInArchive -> ...

--- entries --- [repeats entryCount times, each entry beginning at the general header offsetInArchive]
[[entry]]

-- zip archive footer ---

--- layout of a zip entry ---
entryLength -> ...
entryProperty1 -> zip_file_getProperty1, zip_file_setProperty1
entryProperty2 -> zip_file_getProperty2, zip_file_setProperty2
[..]
entryData -> zip_fopen, zip_fread, zip_fclose