RAM maxed on certain images, fails on higher zoom levels

[kolt54321]

Hi @lovasoa,

Small update before the issue - I know you've been working on dezoomify-rs. I've made some progress on the two open issues on gap-decoder - I believe I can now download images with all metadata embed (viewable from exiftool or similar), and a new naming logic (parses metadata to find artist and date created, to name images as [author + date + image_name (parsing out and excluding author name) + info.image_id + '.jpg']. The new naming logic will make it easier to keep track of downloads by the same artist - a feature I am guessing some would be very interested in.

There is also logic for a batch download feature - not using xargs, as I've forgotten about that easy option - but still working nonetheless. The batch cache doubles as an archive file, so if the command breaks midway, it can pick up right where it left off and not redownload images that were already finished.

That said, I'm still testing these small updates as I'm sure I may have not covered all scenarios. Through testing, I noticed the following link maxed my RAM (32gb) very quickly on Zoom 8, and crashed the script:

https://artsandculture.google.com/asset/the-birth-of-venus-sandro-botticelli/MQEeq50LABEBVg

I also noticed that the currently maintained Dezoomify-rs did not crash on this link, and downloaded it fairly steadily.

Has there been any updates on the Dezoomify side to the actual tile downloading logic? It would be great to continue testing the new naming scheme/metadata and batch files, if some patch could be worked out here. Then we can all move over to Dezoomify-rs if you feel that is the better option.

Edit: This issue may possibly be linked to the max size of a jpeg file - it saved to a png using dezoomify-rs. If 65kx65 is the max a jpeg can take, I believe this on zoom 8 has 108k width.

[lovasoa]

Hello! Awesome to see you are working on improving this tool. You can already open a pull request now, even if your changes are not ready for merging yet, so that we can discuss them.

I recently made a change to dezoomify-rs so that it can stream downloaded tiles to the disk. It creates a png and then appends the pixels to it as they are received from the network. It is a complex piece of code, but it allows downloading images even when they do not fit in memory.

[lovasoa]

The source code for this feature is there: https://github.com/lovasoa/dezoomify-rs/blob/master/src/encoder/pixel_streamer.rs

[kolt54321]

Thanks! Sorry for the delay in responding, not very good at programming here so I don't have high hopes moving the code from rust to python. Very, very cool feature though.

Friend of mine helped me out a bit with the metadata code, I was going crazy trying to find the tags in the page source. I'll open a pull request soon so we can discuss that and the other changes, that's a good idea.

I think the main problem here was that the art piece was larger than JPEG's limit of 65k pixels height/width. Your change to dezoomify-rs hints at another subtle issue; I may not be able to even create a new image in PIL for that size due to RAM limitations - using your original formula of [length x width x 4] = bytes needed, 32gb RAM would just be scraping by, barely.

Focusing on the first issue, I was thinking of a number of options:

1. Put a 65k height/width limit in place and download the next-largest zoom available. I think it's already implicitly available in dezoomify-rs, referencing this here.
2. Just save as PNG, like dezoomify-rs. One downside is that only very new PNG specification allows exifdata (basically: metadata). Given a lot of work (and the naming scheme!) was to add metadata, it wouldn't be able to be embed, so seems like a bit of a shame. Here's an example of embed metadata. Another downside to PNG is that it is only lossless, and the files will always be very large (case in point, above url at zoom 8 png was 10gb when zoom 7 jpeg was <1gb).
3. Use JPEG 2000. Pros - does not have 65k pixel limitation, supposedly "better" compression. Cons - not widely used or supported, decoding is more resource intensive.
4. Use TIFF with JPEG compression. I might have mentioned to you earlier that the original hiccup with using the TIFF format was the 4gb file limit. Using the BigTIFF format and corresponding library dependencies work, but it would mean handling the final image in memory (writing to a numpy array, I think) an additional time - with all the downsides to memory limits. However... ignoring the BigTIFF option, it's possible to use JPEG compression for regular TIFF files, which would (hopefully?) mean that we wouldn't come close to the 2/4gb limit as (I am expecting) the size to be relatively similar to a corresponding JPEG file.

Pros? No 65k pixel limitation (hopefully). Similar JPEG compression quality/file sizes - they even added a quality parameter in PIL 6.1.0 similar to the one we use in PIL JPEG making (though chroma subsampling options are not available). It's all in the documentation here. Cons? Not sure the 65k pixel limit will actually go away, and there are some annoying libtiff dependencies which, in addition to being added to requirements, have to be added to environment PATH as well.

I'm going to see if I can get something working with JPEG-compression in TIFF's. If so, it would seem like a cool alternative to JPEG's when the resolution gets large.

[lovasoa]

What dezoomify-rs does is that when the output file format is not specified, it chooses between jpeg and png depending on the image size.

For very large images, no traditional image format is really adapted though, and a file that will be too large for any image viewer to open isn't really useful. What I've started to implement in dezoomify-rs is re-tiling : saving the output as multiple tiles with an html viewer to view them. You can try this feature in the latest dezoomify-rs builds by choosing an output file name ending with .iiif.

[kolt54321]

Very interesting - and saving as a png is definitely a step up for giving options on images that otherwise would not be able to be saved at all!

I do agree with you that images of this scale are a struggle for image viewers. The intention on my end was printing, and for that higher resolution starts making a difference - but what's the use if the printing company can't open such large files either? Honestly, I don't have a good answer. I hope that in future years processing power will ramp up and allow scans such as these to be viewed, printed, edited etc. Maybe Photoshop can open gigapixel scans without issue, who knows.

Retiling is a great idea, and kudos to you for taking that on. It won't serve printing purposes but should be better on all other fronts.

Edit: Photoshop seems to support 300k by 300k, probably enough even for zoom 9 (if it even exists, haven't come across it yet). At least it's not a total lost cause I guess.

[kolt54321]

Well, small update: JPEG compression in a TIFF file seems to work in theory, but fails in practice. There's a small note that jpeg compression is the better version of tiff_jpeg compression, but that's just a side note. You don't need to actually import libtiff for saving to tiff to work (which is notoriously hard to do, even after manually downloading the dll's and adding them to path), despite the notes mentioning it as a requirement.

With 32gb RAM I was able to create the original file (though it came close to maxing out RAM), but upon the final save it hits me with a JPEGPreEncode: Strip/tile too large for JPEG. error. Whether its treating the entire width/height as one strip, or it's related to this issue, I'm not sure.

At the end of the day the best solution might just be choosing the highest h/w level that fits in a JPEG. I'm sure it would be easier to diagnose with the new framework of building the picture tile by tile, but barring that I'm not sure there's a better way. Saving as TIFF with LZW compression still blows past the 4gb size limit.