RFE: output_format md5_bucket_files

[pbrownrobo]

Right now, there exists "output_format files" and also the calculate hash optional feature.

What would be really nice to see, would be "output_format md5_bucket_files".

eg: file has md5 hash a37f352376.....
So it gets saved to a3/a37f35....jpg

I personally have a post-download conversion routine to organized the downloaded files liie this, but I just realized.... what if img2dataset did it out of the box?

There are multiple advantages to this new feature:

1. the "100,000 files in a single directory" problem mostly goes away
2. collaboration with other people on a dataset becomes a lot easier. For example, if you are both working from a raw image dataset somewhere, one of you can run captioning on it, zip up the directory with just the .txt files, send it over, then the other person can extract it, and they will automatically match up to the right images.

[rom1504]

How many images are you working with ? Storing images as individual files tend to break down for many reasons after 100k or so

…

On Sat, Jan 18, 2025, 14:59 Philip Brown ***@***.***> wrote: Right now, there exists "output_format files" and also the calculate hash optional feature. What would be really nice to see, would be "output_format md5_bucket_files". eg: file has md5 hash a37f352376..... So it gets saved to a3/a37f35....jpg I personally have a post-download conversion routine to organized the downloaded files liie this, but I just realized.... what if img2dataset did it out of the box? There are multiple advantages to this new feature: 1. the "100,000 files in a single directory" problem mostly goes away 2. collaboration with other people on a dataset becomes a lot easier. For example, if you are both working from a raw image dataset somewhere, one of you can run captioning on it, zip up the directory with just the .txt files, send it over, then the other person can extract it, and they will automatically match up to the right images. — Reply to this email directly, view it on GitHub <#448>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437XJ467S5UEGIRMGTT32LJM45AVCNFSM6AAAAABVNUYUMGVHI2DSMVQWIX3LMV43ASLTON2WKOZSG44TMOJZGI2DONY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pbrownrobo]

I'm working with 300k+ files.

The reason I'm working in files format, is that the files then go to AI model training, which requires individual image files
(or at least, my choice of training program requires it)

It also builds a cache of embeddings, which takes hours to build.
If images always have the same filename, then I can do things like

• build one huge tree cache of image files
• move the full tree aside, and create a symlink reference to a subset of the files

In this situation, the training program will preserve its original cache rather than having to rebuild it, which saves me hours of compute time.

[rom1504]

I advise you use a sharded format like webtorrent or tfrecords. Training software can be made much faster when reading such sharded format

…

On Mon, Jan 20, 2025, 02:21 Philip Brown ***@***.***> wrote: I'm working with 300k+ files. The reason I'm working in files format, is that the files then go to AI model training, which requires individual image files (or at least, my choice of training program requires it) It also builds a cache of embeddings, which takes hours to build. If images always have the same filename, then I can do things like - build one huge tree cache of image files - move the full tree aside, and create a symlink reference to a subset of the files In this situation, the training program will preserve its original cache rather than having to rebuild it, which saves me hours of compute time. — Reply to this email directly, view it on GitHub <#448 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437WOVQZJ37X74MESHQD2LRFP3AVCNFSM6AAAAABVNUYUMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBRGEZTQNZXHA> . You are receiving this because you commented.Message ID: ***@***.***>

[pbrownrobo]

I dont think thats valid in this case. Training is entirely gated on GPU throughput, and speed of access for the latent cache.
Switching to those other formats you mention, for the raw images, would not speed it up at all.
Even the process of transforming from raw to latent cache is gated by GPU throughput.

MANAGEMENT of the raw image/cache relationship is much more important. Which is why I'm requesting this change.

BTW: I can understand a view of "this might be useful to you, but I dont have time to code it".

If you agree it could be useful to people like me, then I might code it myself and submit a PR.

[rom1504]

Sure feel free to send a PR.

…

On Mon, Jan 20, 2025, 20:34 Philip Brown ***@***.***> wrote: I dont think thats valid in this case. Training is entirely gated on GPU throughput, and speed of access for the latent cache. Switching to those other formats you mention, for the raw images, would not speed it up at all. Even the process of transforming from raw to latent cache is gated by GPU throughput. MANAGEMENT of the raw image/cache relationship is much more important. Which is why I'm requesting this change. BTW: I can understand a view of "this might be useful to you, but I dont have time to code it". If you agree it could be useful to people like me, then I might code it myself and submit a PR. — Reply to this email directly, view it on GitHub <#448 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437S2K6TASXBESGVPUSD2LVFTRAVCNFSM6AAAAABVNUYUMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBTGEZDINJVHA> . You are receiving this because you commented.Message ID: ***@***.***>