Too much slack

TaozTaoz Posts: 9,941
edited June 6 in The Commons

I think DAZ is giving PAs too much slack.  For example, I just installed a product, using DIM.  1 zip file, about 540 MB.  After installation, total file size 2.1 GB.  Meaning a lot of uncompressed files.

So I checked the zip, and found a number of uncompressed .tif files.  I converted them to .png, and total file size was down at 970 MB (309 MB zipped).

Then I checked a .dsf file, it was uncompressed.  So uncompressed .dsf files are probably responsible for the rest of the bloat.  

If you want to use .tif, at least use LZW or whatever compression (.png compresses better than LZW though and is just as good as .tif in this context).  And compressed .dsf files should be standard.

Are there any tools out there that can rewrite a zip so that uncompressed .tif's are either being compressed (if DS can handle these?), or converted to .png, as well as compressing uncompressed .dsf files, and whatever else that can be done to reduce file sizes?  It will have to update file references in .dsf and other relevant files also, of course. 

 

 

Post edited by Taoz on

Comments

  • richardandtracyrichardandtracy Posts: 5,682

    I thought every now & again DS had issues with .tiff images, so were deprecated anyway. Maybe I'm wrong.

    Regards,

    Richard

  • TaozTaoz Posts: 9,941
    edited June 6

    richardandtracy said:

    I thought every now & again DS had issues with .tiff images, so were deprecated anyway. Maybe I'm wrong.

    Regards,

    Richard

    Well the package is from 2018-06-29 so a bit old, of course.  Whether .tif  are not allowed anymore I can't tell, maybe someone else knows?

    Post edited by Taoz on
  • Richard HaseltineRichard Haseltine Posts: 100,948

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

  • nonesuch00nonesuch00 Posts: 18,131

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

  • CybersoxCybersox Posts: 9,058

    Honestly, the size of files in general has gotten ridiculous.  It's not just that the individual elements are often extremely large and/or uncompressed, but also the fact that everything gets crammed into a single file when it would often actually be more convenient to have the option to not load some parts of a product.  (Example, I'd love it if pose packs NEVER came with a set of expressions bundled into them, as 90 times out of a hundred the expressions just end up bogging down the load time of the base figures.)  

  • TaozTaoz Posts: 9,941

    Richard Haseltine said:

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

    Well I use DIM only, too many issues with Connect and Smart Content.  Making an option in DIM for compressing these files during installation also would be fairly simple I think. 

  • TaozTaoz Posts: 9,941

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

  • NathraiNathrai Posts: 67

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    Not sure why higher resolution was mentioned, both formats support them. But TIFF also supports layers, though usually those are supposed to be provided separately like templates, both because of the hughe file size (not unlike how working with unbaked LIE can hinder performance) and the fact that TIFF has quite a few compatibility problems in many applications.

  • Richard HaseltineRichard Haseltine Posts: 100,948

    Taoz said:

    Richard Haseltine said:

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

    Well I use DIM only, too many issues with Connect and Smart Content.  Making an option in DIM for compressing these files during installation also would be fairly simple I think. 

    You can use the Batch Convert pane to compress files after the fact (which may be what you are doing already).

  • TaozTaoz Posts: 9,941

    Nathrai said:

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    Not sure why higher resolution was mentioned, both formats support them. But TIFF also supports layers, though usually those are supposed to be provided separately like templates, both because of the hughe file size (not unlike how working with unbaked LIE can hinder performance) and the fact that TIFF has quite a few compatibility problems in many applications.

    OK, thanks.   

  • TaozTaoz Posts: 9,941

    Richard Haseltine said:

    Taoz said:

    Richard Haseltine said:

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

    Well I use DIM only, too many issues with Connect and Smart Content.  Making an option in DIM for compressing these files during installation also would be fairly simple I think. 

    You can use the Batch Convert pane to compress files after the fact (which may be what you are doing already).

    Will take a look at that, thanks.  

  • ArtAngelArtAngel Posts: 1,694

    Taoz said:

    Richard Haseltine said:

    Taoz said:

    Richard Haseltine said:

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

    Well I use DIM only, too many issues with Connect and Smart Content.  Making an option in DIM for compressing these files during installation also would be fairly simple I think. 

    You can use the Batch Convert pane to compress files after the fact (which may be what you are doing already).

    Will take a look at that, thanks.  

    TIFF is owned and maintained by Adobe. At 32 bits it holds more data, for depth and other data than PNG at 16 bits. And it has a higher color depth support (but most users do not ultilize 32 bit color and if you don't this is irrelevant. PNG files are much smaller than TIFF files. TIFF supports layers and acts like a PSD file, PNG does not. Not he same but think raw photos. You can save a TIFF over and over again and never lose data.But TIFF files can by HUUUUUUGE.But there was a max file size of 4GB.  PNG can support 16 bits png-48) is open standard meant for the web and is supported by all browsers. It holds 8 bits of data . . . a better version of gif. But if you design a book cover or any other designs intended for print and not the web, you will be sadly dissapointed at the print quality. The colors will differ.  and compare a PNG 

    Basically TIFF shines for printed products CMYK (like printed t-Shirts Calendars Book covers) etc., while PNG shines for webpages, youtube & social media headers and instagram pics.,basically any online images, anything web related because it utilizes RGB and is a much smaller file.

  • AnoAno Posts: 1
    edited June 8

    If you're not already doing so, have you tried to use Windows' built-in file compression for your runtime directory (set to "Compress to save disk space" via it's proprties' "Advance Attributes" setting)?

     

     

    Post edited by Ano on
  • Richard HaseltineRichard Haseltine Posts: 100,948

    ArtAngel said:

    Taoz said:

    Richard Haseltine said:

    Taoz said:

    Richard Haseltine said:

    Iray had a spell of not handling Tiff  well.

    If you isntall through Connect (Daz Studio itself) the .duf and .dsf files will always be compressed.

    Well I use DIM only, too many issues with Connect and Smart Content.  Making an option in DIM for compressing these files during installation also would be fairly simple I think. 

    You can use the Batch Convert pane to compress files after the fact (which may be what you are doing already).

    Will take a look at that, thanks.  

    TIFF is owned and maintained by Adobe.

    Tiff is, like FBX, somewhat variable in its structure. Photoshop always had special code to try to cope with the variations but other applications (inlcuding Illustrator) could fail to read a Tiff from a diferent applications.

    At 32 bits it holds more data, for depth and other data than PNG at 16 bits. And it has a higher color depth support (but most users do not ultilize 32 bit color and if you don't this is irrelevant. PNG files are much smaller than TIFF files. TIFF supports layers and acts like a PSD file, PNG does not. Not he same but think raw photos. You can save a TIFF over and over again and never lose data.

    PNG is also lossless.

    But TIFF files can by HUUUUUUGE.But there was a max file size of 4GB.  PNG can support 16 bits png-48) is open standard meant for the web and is supported by all browsers. It holds 8 bits of data . . . a better version of gif.

    Nort sure what you mean here, but PNG is not comparable to Gif other than supporting transparency (albeit in a different manner, a prioper alpha channel)

    But if you design a book cover or any other designs intended for print and not the web, you will be sadly dissapointed at the print quality. The colors will differ.  and compare a PNG 

    That sounds like a colour management/tagging issue.

    Basically TIFF shines for printed products CMYK (like printed t-Shirts Calendars Book covers) etc., while PNG shines for webpages, youtube & social media headers and instagram pics.,basically any online images, anything web related because it utilizes RGB and is a much smaller file.

  • PerttiAPerttiA Posts: 10,024

    Ano said:

    If you're not already doing so, have you tried to use Windows' built-in file compression for your runtime directory (set to "Compress to save disk space" via it's proprties' "Advance Attributes" setting)?

    I would not touch that with a ten foot pole 

  • TaozTaoz Posts: 9,941
    edited June 8

    ArtAngel said:

     

    TIFF is owned and maintained by Adobe. At 32 bits it holds more data, for depth and other data than PNG at 16 bits. And it has a higher color depth support (but most users do not ultilize 32 bit color and if you don't this is irrelevant. PNG files are much smaller than TIFF files. TIFF supports layers and acts like a PSD file, PNG does not. Not he same but think raw photos. You can save a TIFF over and over again and never lose data.But TIFF files can by HUUUUUUGE.But there was a max file size of 4GB.  PNG can support 16 bits png-48) is open standard meant for the web and is supported by all browsers. It holds 8 bits of data . . . a better version of gif. But if you design a book cover or any other designs intended for print and not the web, you will be sadly dissapointed at the print quality. The colors will differ.  and compare a PNG 

    Basically TIFF shines for printed products CMYK (like printed t-Shirts Calendars Book covers) etc., while PNG shines for webpages, youtube & social media headers and instagram pics.,basically any online images, anything web related because it utilizes RGB and is a much smaller file.

    Well, I could write a long post about the info I've found as well as the experiments I've made with TIFF compression, conversion between TIFF and PNG and reverse etc., since I started this thread, but unfortunately I don't have the time.  It seems that the more you study all this, the more complex and subjective the whole thing becomes.  So just a few interesting points I've come across:

    "The color space which the human eye perceives has its upper bound at 10 million colors. Anything beyond that is not really distinguishable to the human eye, but will still appear more colorful when being processed by the brain."

    https://medium.com/hd-pro/color-bit-depth-and-perception-in-human-vision-ca97313722d3#:~:text=The color space which the,being processed by the brain.

    So the potential benefits of creating images on screen or printed with a color space above say 16 million colors seems pretty marginal to me, if noticeable at all?  As for the "still appear more colorful when being processed by the brain", that sounds a bit subjective.  The mind can process the same sensual data in a wide range of different subjective ways, depending on secondary factors. 

    Also:

    "While it is generally true that CMYK is better suited for printing due to its ability to accurately reproduce colors on physical media, there are situations where an RGB image might look better when printed. Here are a few reasons why this might be the case:

    1. Color Gamut: RGB has a wider color gamut compared to CMYK. This means that RGB can represent a broader range of colors than CMYK. If your image contains colors that fall outside the CMYK color gamut, they may not be accurately reproduced when converted to CMYK for printing. In such cases, the RGB version may look better because it retains those out-of-gamut colors.
    2. Color Management: The way color management is handled during the printing process can affect the final output. If the color profiles are not set up correctly or if there are inconsistencies in color management between your digital file and the printing process, the colors may not be accurately reproduced in CMYK. This can lead to differences in color appearance between the RGB and CMYK versions.
    3. Monitor Calibration: The appearance of colors on your computer screen (RGB) can be affected by monitor calibration settings. If your monitor is not properly calibrated, the colors you see on screen may not accurately reflect how they will appear in print. This can lead to discrepancies between the RGB preview and the final printed output.
    4. Paper and Ink Quality: The quality of the paper and ink used for printing can also impact how colors appear in the final print. Some papers may not be able to accurately reproduce certain colors in CMYK, leading to differences in color appearance between the RGB and CMYK versions."

    (long and interesting thread):  https://www.quora.com/They-say-that-CMYK-is-better-and-more-accurate-in-prints-rather-than-RGB-and-I-get-why-But-for-some-reason-the-print-I-made-looks-better-in-RGB-Why-is-that

    So all in all there seem to be so many factors and so much subjectivity involved that I'll just stick to my current personal viewpoint which I probably share with many others:  does it look good, it's OK, does it not, modify it until it looks good.  wink   At least when it comes to art - in science and technical contexts more strict rules for exact color handling and detection may be necessary of course. 

    As for TIFF I'm not sure what possibilities DAZ Studio offers here, if any.  Maybe it can be used when working with Canvas/layers, to keep the layers separate when saving the render?  Not something I have any experience with anyway.  I can see though that DS is using LZW compression when saving "normal" renders in TIFF format which, as far as I can tell from my experiments, reduces the color space to 16 million (no layers, AFAIU).  Whether there is an option for saving them uncompressed I don't know. 

     

    Post edited by Taoz on
  • felisfelis Posts: 4,338

    As I see it, when working with canvases, you should save them as exr.

  • TaozTaoz Posts: 9,941

    Ano said:

    If you're not already doing so, have you tried to use Windows' built-in file compression for your runtime directory (set to "Compress to save disk space" via it's proprties' "Advance Attributes" setting)?

    Thanks, but having to decompress every single file whenever it's being used (presuming that's what it does), will affect performance. 

    I'll try using the Batch compressor in DS which Richard mentioned, it only compresses relevant files, and what's left that isn't already compressed is probably not significant.

  • lou_harperlou_harper Posts: 1,163

    There is zero reason why image files in DAZ should be TIF. Even PNG higher quality over JPG is mostly theoretical. By wich I mean it't technically true, but you won't see the difference with the naked eye.

    And CMYK used in print because print uses actual physical ink, and is subject to the physical limitations of the process. I.e. smaller color range. Deep black, and very saturated colors tend to suffer the most.

  • TaozTaoz Posts: 9,941

    felis said:

    As I see it, when working with canvases, you should save them as exr.

    OK. 

  • TaozTaoz Posts: 9,941

    lou_harper said:

    There is zero reason why image files in DAZ should be TIF. Even PNG higher quality over JPG is mostly theoretical. By wich I mean it't technically true, but you won't see the difference with the naked eye.

    And CMYK used in print because print uses actual physical ink, and is subject to the physical limitations of the process. I.e. smaller color range. Deep black, and very saturated colors tend to suffer the most.

    In general, hitting the exact right color among say 16 million by mixing 3 or 4 colors seems pretty spectacular to me, yes.

  • PancakePancake Posts: 69

    In terms of visual quality, I've noticed a lot of textures have issues, particularly in the anatomical elements - even on daz originals. Freya 9's M gens was the first one I noticed it with. There's a little triangle hidden under the peen where the textures didn't match up right. Haven't checked if this has been fixed admittedly - but issues with the gens textures not matching up right or being artistically different from the rest of the skin are pretty common to pass quality control.

  • nonesuch00nonesuch00 Posts: 18,131

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    I haven't used Photoshop in a long time but editing an full resolution image with every bit of the data supplied by the raw original image used to require tif, which is more or less a standardized raw image format. I'm not sure if png, even if uncompressed, has all the data structures and raw data to allow all the manipulations of an image that tif allows.

  • Richard HaseltineRichard Haseltine Posts: 100,948

    nonesuch00 said:

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    I haven't used Photoshop in a long time but editing an full resolution image with every bit of the data supplied by the raw original image used to require tif, which is more or less a standardized raw image format. I'm not sure if png, even if uncompressed, has all the data structures and raw data to allow all the manipulations of an image that tif allows.

    Digital images from a camera should ideally be in CR2 or anothr raw format that just has the sensor data with no processing.

  • TaozTaoz Posts: 9,941

    nonesuch00 said:

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    I haven't used Photoshop in a long time but editing an full resolution image with every bit of the data supplied by the raw original image used to require tif, which is more or less a standardized raw image format. I'm not sure if png, even if uncompressed, has all the data structures and raw data to allow all the manipulations of an image that tif allows.

    As far as I understand, PNG doesn't support layers like TIFF does, but it does support up to 48 bit colors (16 bit per channel).  

  • TorquinoxTorquinox Posts: 3,335
    edited June 9

    For the most part, high-quailty jpeg (no visible artifacts) has been sufficient for my needs with the occasional png when I need opacity channel in the image. Now, exr would be quite helpful for DS and other 3D, except that many image programs don't seem to handle exr files very well, if at all.

    Post edited by Torquinox on
  • nonesuch00nonesuch00 Posts: 18,131

    Taoz said:

    nonesuch00 said:

    Taoz said:

    nonesuch00 said:

    tif are a favor by the PAs for those customers that want to edit the textures at higher resolution.

    Can you do anything with .tif you can't do with .png?  I've tried to find some info about it, but not found any clear answers.

    I haven't used Photoshop in a long time but editing an full resolution image with every bit of the data supplied by the raw original image used to require tif, which is more or less a standardized raw image format. I'm not sure if png, even if uncompressed, has all the data structures and raw data to allow all the manipulations of an image that tif allows.

    As far as I understand, PNG doesn't support layers like TIFF does, but it does support up to 48 bit colors (16 bit per channel).  

    Thanks for the clarification. I was told always save images you want to be able to edit in the future in tif format.

  • TaozTaoz Posts: 9,941
    edited June 9

    nonesuch00 said:

    Thanks for the clarification. I was told always save images you want to be able to edit in the future in tif format.

    Well if you want to have a source file with as many potential editing and conversion options as possible, that also seems to be the best choice (whether there are other similar or even better formats I don't know).  

    Post edited by Taoz on
Sign In or Register to comment.