Daz Studio compressed File Format
joelhacker
Posts: 10
I've made an Applescript that compresses or decompresses (users choice) Poser format files and thought it might be nice to have it work on DS content as well. Does DS use regular zip or some other compression tool? What are the list of compressed file extensions and list of uncompressed file extensions? I'm not that familiar with DS.
Any help greatly appreciated
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
DS uses gzip compression.
There are no specific file extensions for compressed DS files, they use the same as uncompressed files.
Poser also uses gzip, but DS not having separate file extensions does complicate matters. I'll have to think about whether it's worth it to include that functionality since DS has batch compression built in. Thank you
DS uses gzip. The only way to tell if a DS file is gzipped is by the gzip magic number, which is a little endian 0x8b1f as the first two bytes of the file.
Thanks, idk why I didn't think of that. Gzipped files that I gzipped on my Mac from the terminal have a magic number of 1f 8b, not 8b 1f. Do DAZ files have a different endianness on their compression? I'm not at all familiar with DS, I find the user interface clunky, confusing, and crowded, but I downloaded it. How do I compress DS content via DS Studio to verify that my app will handle them properly?
btw, I CAN just use the file command, but I still need examples of each type of ds studio file compressed by ds studio itself to make sure everything works properly. File command returns a result of ascii text on uncompressed duf files, but I'd want an example of a ds file compressed by ds itself to make sure that file command can accurately recognize it as a compressed file. So if someone could help a noob compress files within DS that'd be great. My googling is not helping
Your help is really appreciated.
Don't know if this option just isn't on the mac version, but in the windows version, there's the "Batch convert" tab that does exactly what you're talking about doing with your script.
It allows the user to decompress or compress either individual files, or entire directories.
The only two types that this applies to are DSF(morphs/geometry/uv/etc) and DUF(user facing files).
Most other files, DSA/DSB/DSE, DJL, DHDM, are going to either be encrypted, proprietary, or can't be compressed/uncompressed.
Found the batch converter window. Was able to confirm that the linux command 'file' properly recognizes compressed DS files as compressed files and uncompressed ones as ASCII text. You've all been very, very helpful. Thanks so much!
The applescript app that works on all Poser format files is available here
https://sourceforge.net/projects/macposerutils/files/PoserSqueezer/
I should be able to add DS format files sometime next week. For Mac OS X only, only tested on Monterey with an M1 processor but it should run on Intel macs with os x version 10.7 and up. I think. Anyone wanting to give me feedback can email me at joelhacker at icloud dot com
Not sure what you can do in AppleScript; in Python I just try to open it as a gzip file first and see if it succedes.
Problem is I believe you can actually gzip a file that is already gzipped, resulting in wasted resources. I'd rather know for sure whether a file is compressed or not before calling gzip, but thank you. Luckily the UNIX command 'file' seems to properly identify DS files as compressed or not, so there's no problem other than the time to actually just write up the script. Probably get to it middle or end of next week.
@joelhacker
That's as expected. "Little Endian" means that the less-significant-bytes are stored in lower addresses. Unlike the old PPC processors in older Macs that had programmable endian-ness, or even older Motorola 68K based Macs that were Big Endian, Intel processors are 100% Little Endian. But western languages and the programming languges they invent are left-to-right and write the mort-significant-bytes first. So in a programming language where you refer to a 16-bit constant such as 0x8b1f, this is actually stored in sequential devices (e.g. memory or hard drives) with the bytes in reverse order with respect to that constant's orthographic representation: 0x1f 0x8b.
All this is because back in the day, memory access was so slow that the processor had to present the column address to the memory controller before the row address so that it could start decoding the requested memory address sooner.
I don't know what Applescript is, but I will suggest using a language that has some notion of objects, so you could write a function that accepts a file path and returns an abstract object that represents an open file that can be read, hiding the details of whether the file was gzipped or not. And also that has some sort of destructor that automatically closes the file and deletes the uncompressed temp file if there was any, when the file object goes out of scope, and is exception-safe. I respectfully offer that you probably shouldn't be taking seriously any programming language that doesn't support some type of abstraction like this. C++ and Python certainly do.
Only the GUI is in Applescript. The majority of the work is through the UNIX command line. Applescript has a very convenient "Do shell script" function built in that allows me to have zsh do all the heavy lifting while Applescript automatically handles the UI elements for me. I can drag and drop file and folders onto the app, I can do a dialog window, I can choose files and folders, all with Applescript then hand off the actual process of compression and decompression to the UNIX command line. If I was making a product to sell I would absolutely make it with Java, Python, or Objective C, and probably make it cross platform, but doing it this way is much, much faster and I've only at the last moment decided to include functionality for DS file types.
Have I gone blind, or has the option to save a scene uncompressed disappeared from 4.16???
Scenes do not (yet, at least) have an option to save without compression. A Scene Subset does.
See http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/file_io/save_duf_scene/start
You might try using gunzip on a duf file youtself, rename it back to have a .duf extension, and cross your fingers and hope that the DS code recognizes whether it is compressed or not. That the option to compress or not exists in certain cases implies that that might indeed be the case.
Duf files can be uncompressed by opening them in 7-Zip with "Open Archive", no need to change the extension
Thanks all for the comments. Easiest seems to be to use the D|S batch converter. This has the advantage that it will decompress what it can from a damaged .duf also; which 7-zip, winzip etc won't process. I never tried gunzip on the damaged file as getting it loaded up looked too fiddly.
The message to all is to take manual backup copies of important files every so often. Trying to recover this compress-damaged .duf of mine is taking hours of experiementation and not making much progress. Findings being reported here as they are made. https://www.daz3d.com/forums/discussion/367831/how-to-fix-a-damaged-duf-file
And just for the record, the desire to decompress is more a desire to never compress in the first place given several bad experiences with files that die in compression. My time trying to work around a failure is way more valuable than a few lousy kilobytes of disk space saved.
I know I'm, like, really late to your game, but just a thought:
Instead of messing around looking for a magic number and worrying what it is, an worring about specific cases for different applications, why not just read up to the first non-whitespace byte (probably the actual first byte) and see if it's "{"?
I mean, .duf anf .dsf and so on are all JSON.
And while PZ3/CR2/etc files are not JSON, they also start with the same character because Poser markup is also curly brace deimited (though stricter in whitespace use-- Poser will not accept de-carriage-returned files for instance)
But GZipped files never will ever start with "{" AFIACT.
The only files this leaves out that I can think of is .obz files and those are easy to tell by looking at the extension.
for the record, you can call:
to extract the first character for analysis
Because it's probably better to predicate the correct functioning of your application on an old, well-known, and agreed-upon convention like the magic number, rather that something that is AFAICT. It's not supposed to be easy, or simple, it's supposed to always work, and this could be the difference between an app that is rock-solid and one that is flaky. Don't inject new conventions or research unless it's absolutely necessary or allow yourself to be seduced by simplicity just to be drop-kicked in the teeth by some future change that voids one of your assumptions and causes your app to not work.
I confess that your approach will most likely work (I have to invent some unlikely scenarios to break it, like changes to the JSON spec or a non-ASCII encoding which might even be prohibited), but it's not what a software engineer would do because it does add ways in which an app can misbehave. There are only two possibilities for endianness, and there are facilities to make it transparent on every platform.
I'd also suggest not depending on the file extension. There's a long history in the UNIX world on why this is not really a good idea, first among them being that there's no way for the app to control them and a user could change it, even inadvertently. Under the hood it's the same operating principal: don't embrace simplicity, but rather reliability.