Genesis Bloat and End of File Error
Mosk the Scribe
Posts: 888
I have several files - each of them just over 4GB - that give me the following message when I try to open them:
File c:dazdevel/hudson/hudson_home/jobs/nb_carrara_85/build/carrara85rscmccoresourcecommonMCToken.cpp
An end of file error occurred while executing open
These scenes each have 2 genesis characters - which is what causes the enormous file size.
I have not been saving these as compressed. And while I do save versions on some, since each is 4GB, I can only make so many backup copies before filling up my drive.
Is there anything I can do to retrieve these files or do I need to just consider them corrupted and deal with it?
(I've already tried the trick of renaming as a .car.gz and extracting with 7zip -but as noted, these were NOT compressed when saving so that didn't work)
Is there a 4GB cutoff that leads to the End of File Error?
Any way to significantly reduce Genesis so scenes aren't so bloated? This has become an immensely frustrating problem. (I need to use genesis, because I use it in conjunction with FaceGen)
Thanks for any suggestions.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
I have lately been compressing my files without issues
however I usually load a fresh Genesis etc customised in DAZ studio and just save my Carrara shader tree and NLA animations
2GB is usually the 32-bit file size limit, and older versions of NFS have a 4GB limit, beyond that it will be OS and application specific.
I have Windows 10 64 bit with 64 bit version of Carrara 8.5 Pro - but I've now had multiple files over 4GB that refuse to load with same error message, costing hours of work and causing endless frustration!
Does Genesis cause this huge file bloat for everyone - or is there another approach I can try, because this is a constant issue that really hampers my ability to use genesis, which I would really like to use for a variety of reasons.
Yep, I get that also
i reverted to splitting large scenes like that into several files and compositing in PSE
There is a way that you can possibly patch up the file so that it opens, if you saved it uncompressed and have a text editor that can handle files that big. (I use TextWrangler on the Mac, but someone hopefully can advise a suitible editor for Windows)
I discovered this issue when using a program called P3DO to change the internal icons of prop files. You can see my initial findings here: https://www.daz3d.com/forums/discussion/comment/2354921/#Comment_2354921
Precis: There is a tag in the .car file called Vlst. It's followed by a stream of comma separated numbers, the first of which is the the count of those that follow. This list can get very long, and I suspect that occasionally it overflows the size of the write buffer and gets truncated. So the number of items in the list doesn't match the count, and this causes the Unexpected End of File error on loading.
If the data in that tag is just a string of "-1," I don't think it matters particularly how long it is. I've never encountered a problem from just copy & pasting another instance of the tag from elsewhere in the file. I haven't come across a file (yet) that had different values. Of course there may be more than one broken tag, and there's no guarantee even that this is the same problem, or that there isn't some 4GB limit being hit somewhere (you're not using 32-bit Carrara by any chance?).
Still, if this helps you get the file to open, you can then save it out in smaller chunks, which will help for next time. (You can import the various chunks that make up your scene, there's no *need* to render each chunk separately, even though it can sometimes be useful to do that.)
The Genesis bloat, I suspect is caused by the number of different character morphs that have been added to it. Because of the way that Carrara stores its data, they'll all be pulled in and added to the file. I don't know if there's a way to create a genesis figure that only has a single character morph. @Mistara sounds like this might be your area of expertise. Any thoughts?
Hey @TangoAlpha (and @Mistara ) -
Thanks for suggestions. I was able to open the .car file in Dreamweaver, but could not find any Vlst tag.
In terms of Genesis bloat - I do have lots of my own morphs built into carrara from creating faces via facegen (which then gets stored inside genesis) - and have a bunch of other morphs from DAZ PA's. Like the functionality of having everything available in a single character - but given the problems that's causing, would building Genesis figures which only contain the particular morphs I need for a given project might be a workable solution if the process isn't too cumbersome and it significantly reduces the figure size so I can have more than one figure in a scene and not end up with crashes and unopenable files.
Usually, this kind of error happen when there's a lack of closing braces compared the the opened ones.
trying to find out how much are missing in a 4gb file may be cumbersome, but you can try adding a dozen of them at the end of the file (or more if necessary). Missing information will be lost all the same but, if you can open your file, maybe you can salvage enough of your scene.
We've all been saying this for a long time, but I don't think this is true.
I tried loading Genesis 1 and 2 from clean installs on a new computer (where DAZ Studio and Carrara had never been installed before). I only had the base figures on that machine, with none of the other content installed and they still caused incredble bloat and slow save/load times in a Carrara scene. I compared the file sizes and save/load times on that computer v. my main computer and there was very little difference despite having hundreds on morphs on one and none on the other.
I don't know what it is, but there is something different about the Genesis figures themselves, v. previous figures, that is causing this.
I do exactly this; create a new obj out of the standard Genesis, import it into Studio, use the Transfer Utility to rig it the same as Genesis, then save whatever morphs from the main Genesis I'm going to use and load them into my new Genesis I just created. Then I have a 'Genesis lite' that has only the morphs I need to use for that particular figure, they save quick to my objects tab in Carrara and load very quickly too, low on memory when saving scenes. For example, I picked up 'The Ghoul' for Genesis a couple of days ago. I don't really need 5000 different morphs for a ghoul character, it's not like I need phonoemes for example. I made a new 'lite' ghoul character with basic female/basic male morphs (so I could have it be either sex) added in a pot belly morph and an emaciated morph, then a few basic expression morphs (anger, rage, afraid, surprise) as well as mouth open and mouth open wide. I think all told it was like 10 morphs total, then I saved it out as a figure/prop preset, imported into Carrara, and saved to my object tab. Now I've got a ghoul that loads almost instantly, takes no time at all to save, can wear any Genesis clothing and poses like Genesis does, and is ready to go at any time.
Hey Jonstark - When you load that lite version of Genesis without anything else, how big is your Carrara file? (I'm running into cases where I'm over 3GB as soon as i add 2 genesis characters, dress them and add hair - and that means that with just a little animation and other items for the scene, i quickly run into trouble.
Assuming your method gets you a small file size (reflected in the almost instant loading / saving), can you point me to some resources that spell out exactly what i'd need to do (to rig in DAZ and only incorporate the morphs i want)
Thanks
this looks like it is worth a try, TY Jonstark
times two :)
good tips to try out
Well shoot, something's going weird. I saved out a single scene with not much in it, a light and an infinite plane and my Ghoul that I had made with only 12 or so morphs Genesis rigging that I had saved to my objects tab. This makes no sense to me, because the thing loads super fast (like maybe a few seconds) and saves to the objects tab equally quickly, and when I save the scene itself it takes a little longer but not much longer, plus I just did a scene with a ton of these ghouls all over the place and there's no slowdown and even that larger scene doesn't take all that long to save and load, but when I checked the size of the .car file for the single Ghoul by himself it's much bigger than it should be, around 165 mb. I can't explain that at all, it's got me scratching my head and wondering what it is I'm missing here... I hope I have steered anyone towards fool's gold, this is a new thing I've been doing lately (ever since I figured out how to create a Genesis with geografted anatomy where the textures are correct, it's essentially the same method, which got me thinking about how to make a Genesis clone that had far fewer morphs and was purpose built). My point being, I *thought* this was a good viable solution, and it certainly seems to be in the way the scenes behave, save and open, but I can't figure out why the .car file would be so large...
(I'll post the scene I'm talking about with all the ghouls in the Cripeman challenge thread, wouldn't make sense to post the same pic in 2 different threads)
Hey Jonstark-
Just loaded my current regular genesis file, which has lots of morphs - and that 'only' takes 365MB and loads and saves fairly quickly.
365MB may seem large, but this is in comparison to the 3GB that I get when I load a single Genesis figure which has been given clothes, hair, and some other things attached (DAZ Anatomy, DAZ Skeleton).
So while Genesis may be 'bloated', my real problem must be with the various items I've added for a pair of characters
Too far into the project to rebuild that figure for now, but is there a simple way to automatically limit texture size for all items in scene to see if that makes a significant difference?
Now, if I add my two characters (3GB each) I end up with a 4.2GB file, which I am not able to open (get End of File error)
I'm trying to save a compressed version of file now. First time I tried it, Carrara became non-responsive. So far on this attempt it's been about 10 minutes, and sitting at 22.9% compressed rightnow but not frozen.
Would really like to understand what the problem is and potential workarounds as this has been complicating my work quite a bit!
Thanks for any help / suggestions.
maybe genesis is adding all its morphs and targets to the clothes as well
we making genesis characters and clothes and hair into standakone figures?
jcms and everything?
I got tired of the issue with geografted anatomy for Genesis1 and drilled down to try to find some solution (even though I don't really need close up shots of genitals for any real purpose other than irked me that this was a limitation even with Genesis1, the best-supported of all the Genesis figures by Carrara).
After experimenting, found I could attach the anatomy in Studio, export the .obj out, re-import it back in, use the transfer utility to give the .obj I created the same rigging as the original Genesis1 (sort of like making it into a 'clothing' version of itself) and then save that as a figure that I could open in Carrara that would have all of Genesis1 rigging yet wouldn't go crazy on the geografted parts (because nothing is geografted anymore, it's just a whole different mesh that includes the anatomical elements). Of course this new figure wouldn't have any morphs that the original real Genesis1 had, so I started making morphs from my original Gen1 with the geografts attached, saving them, then using the morph loader pro to put the morphs on the new Genesis clone had made that had the anatomy as part of the mesh instead of a geograft. Seemed to work very well, but as I'm saving all these morphs I'm thinking 'wait, I don't use even a 5th of these in my real work' and it occurred to me I could simply make various Genesis1 characters with only the morphs I actually use.
I don't know about JCM's though, I hadn't even thought of them to be honest. I don't know if that's a part of what is copied over using the transfer utility or whether that's completely separate. My Genesis1 clones seem to work fine, all the bends are the same etc in Carrara as my original Genesis. I don't know whether that means the JCM's are somehow also copied over as part of the transfer utility or whether Genesis doesn't really depend much on JCMs and it's all down to the weight painting of the rigging, but either way what I care most about is that it seems to work fine with no discernable differences to the origina figure.
I also found that using this same method, with one extra step of changing the rigging to 'blended' as you suggested in a different thread I can get a Genesis3 clone into Carrara that seems to work just as well as the original. Although this doesn't seem to work for Genesis8, however I gathered you figured out the problem on that one already and already devised a solution :)
we were trying the clone transfer last year, shout out to SileneUK and Vyusur
,
transfer wasnt bringing over the rotation limits.
saved as a product, lost it's connection to the morphs folder, which seems what we want for a smaller memory footprint for Gn1,
without jcms, bends looked off.
was some pitfalls
for this project, are we dragging genesis1 characters to the carrara tray as carrara figures?
Yup, I was saving to the objects tab as I went, I always find things load faster and are more stable if they've been turned into carrara objects. I'm only a little while into doing this, probably lots of pitfalls I didn't run into yet, I hadn't realized rotation limits weren't carried across, though I'm not surprised. As far as bends, when I first started to play with this, I would bring up both the original Gen1 and the clone I made in the same space, applying the same pose to each, slightly different coloring for both so I could what was where, every pose I tried exactly matched, so I assumed all was well after running through a gamut of different poses. Perhaps though more extreme poses might reveal the bend problems/differences.
Vyusur is a clothing fitting expert! I just faff around and get lucky sometimes.
By a happy accident I was making G3M clothing fit my Misty Janette by fitting it to Janette in DS, then doing the transfer thingy, then exporting as support asset. Steps I laid out in a document to be sure not to miss anything. I am careful to save it to the same file as the original clothing item so I know what I have already re-rigged. Once in Car, I fitted it to my over-muscled Neanderthal woman... with a bit of tweaking with the magnet tool for breast coverage, then saved as a Car file.
You are right, Jon... IMHO kt-bashed rejigged clothing does fit better and faster once a car file. Even when textured with lots of Car hair!
My next challenge is to try this with some previous generation clothing... it will be interesting. If I can do it with a minimum amount of pain (eg cross-gender fitting) then my characters will have a whole new wardrobe. I also am thinking I can do with with G3 stuff to fit some older G2 characters I have.
I keep thinking I want to make a video tut on how to fit clothing from one generation to another, so far it seems a very simple process. The clones certainly make it easier, but I don't even think they're necessary really, just export the clothing as an .obj, then re import back into studio, put it roughly in the right place on whatever character you want it to fit to, then transfer utility to put the rigging for that character into the clothing object. Save as support object, then open the new clothing in Carrara, fit it to your Genesis, unlock the topology and use a bit of soft select while modeling in the assembly room and pull the various bits into the right places, and voila done deal. Actually writing it out makes it sound more complicated than it really is, it's pretty simple :) I figure I paid real cashy money for lots of clothng, I'm going to make sure I can use all of it on whatever figure I want. The only thing I can't find a workaround for is clothing sets from very early generations that include stuf that isn't clothing (specifically 2nd skin textures instead of real clothing objects, that can be a bummer).
HA!!! I was lost halfway through the 2nd sentence. If you ever do a video, however, I will watch. Making clothing compatible (and all this exporting-importing in various formats) has not been a priority for me, as I am still too busy learning basic Carrara functions. But I'm sure that I will need to know this eventually.
This video covers a very simple object by Mike Moir, but it worked for me. The Transfer Utility part is skimmed over a bit as he made a head-dress item which he modelled in Carrara.
For kit-bashing...I have now put a G1 boots item on my Janette's feet in DS and that is where I started... did the TU operation, removed the Janette figure afterwards, leaving only the boots in the DS scene, then saved them as a Support Asset in a handy file (for me that's the same place the item lives for use in DS as under Content in Carrara mode, I know where to find it and it's properly attached to its textures. Then, into Carrara and Bob's Your Uncle!
Hope to try some V4/5 or M4/5 items next.