Why Does It Take So Long To Load Figures? ("Solved", with GUIDE)

mindsong

TheMysteryIsThePoint said:

thenoobducky said:

TheMysteryIsThePoint said:

thenoobducky said:

TheMysteryIsThePoint said:

Richard Haseltine said:

I will repeat this one more time: I am not denying that loading can get painfully slow, I am disputing your assumption - and it is an assumption - that this means bad architecture, unless you are making the trivial point that since the process can get slow then it is a process prome to slowness without any implication that there might have been much better ways to achieve the same end. That doesn't, of course, mean that there definitely aren't better ways to achieve the same end - but neither of us is in a position to know, and therefore neither of us is in a postion to point fingers.

You are wrong about that, as well. Here is why: On one extreme, we have what the system appears to be doing: at least parsing lots of morphs that do not deform any mesh. On the other extreme, a node's duf file contains references to only the morphs used, and those files in turn have all the vertex offsets necessary to know how to deform the mesh. DS's source code may be locked away, but anyone can read the DSON spec as well as see it in action in an actual .duf file for themselves and protect themselves against false statements like the one you made above.

Except that is not the root cause of the problem. The biggest slowdown seems to be caused by Daz needing to create every slider in shape and posing pane with their associated erc formulas. It observation come from looking at filesystem activity when Daz load a figure and looking at what it actually read. 

The slider is a desired feature because it is beginner-friendly and many people want it. This is inherently an O(n) process. Daz needs to go through every file to get the sliders. Maybe Daz can optimise the process to be more efficient, or making it multi-threaded. But multi-threading would only buy you 2-5x performance improvement, considering most CPUs only have 4-8 cores. People have suggested only parsing the file to create dummy sliders and formulas and create the actual formulas as needed. But it is not clear to me that not creating the actual formula would be that much faster since the file still needs to be parsed completely.

Seems to me that the only solution is to go outside of the box and change the requirements. What Daz really needs is a way for the user to specify that it should load all morphs from a list of folders, all morphs from a list of product, and allow the user selectively loading morphs from the rest later.

It seems like there are many, many things that could be done, just none of them are being done.

Lets discuss what are those "many, many things" and which option you would prefer. Although given the dev's stated priority in the DS5 thread being getting the program out as soon as possible, I wouldn't hold hope for change until DS5 is released and bug fixed.

I was referring to you yourself, and the suggestions that Richard Haseltine suggested. But my argument is not that I know better than the devs, or that I can have a precise opinion on the subject of code that I have never seen. My position is primarily that it is not reasonable to think that the way things are now is simply the best they can possibly be, rather than the result of an architecture that was sufficient under older circumstances that is not sufficient now, because complex systems always present more opportunities for tradeoffs than professionals will have time to analyze and implement.

These guys wrote and/or maintain Daz Studio. I'm sure they know what they are doing. It is just not reasonable to state that a certain implementation is the best possible implementation just because no one can demonstrate something better, an especially foregone argument for a closed-source app. Personally, a large portion of things I have ever written over a three decade career are things that I would have done differently if I had understood the requirements better or if they hadn't changed, had more domain knowledge of the business, had a PM that would budget for test-driven development, or had just read a certain online article sooner. I did not anticipate how people would react so viscerally to the word "bad", even when objectively used. Heck, by my own definition, I write "bad" code all the time... that is the rule rather than the exception.

nicely put. (again :)

(and people like me think your 'bad' code is pretty darned good, so that just adds all kinds of fun to the situation, doesn't it?)

heh,

--ms

TheMysteryIsThePoint

mindsong said:

nicely put. (again :)

(and people like me think your 'bad' code is pretty darned good, so that just adds all kinds of fun to the situation, doesn't it?)

heh,

--ms

Thanks, MS. Words like that keep me going.

thenoobducky

TheMysteryIsThePoint said:

thenoobducky said:

TheMysteryIsThePoint said:

thenoobducky said:

TheMysteryIsThePoint said:

Richard Haseltine said:

I will repeat this one more time: I am not denying that loading can get painfully slow, I am disputing your assumption - and it is an assumption - that this means bad architecture, unless you are making the trivial point that since the process can get slow then it is a process prome to slowness without any implication that there might have been much better ways to achieve the same end. That doesn't, of course, mean that there definitely aren't better ways to achieve the same end - but neither of us is in a position to know, and therefore neither of us is in a postion to point fingers.

You are wrong about that, as well. Here is why: On one extreme, we have what the system appears to be doing: at least parsing lots of morphs that do not deform any mesh. On the other extreme, a node's duf file contains references to only the morphs used, and those files in turn have all the vertex offsets necessary to know how to deform the mesh. DS's source code may be locked away, but anyone can read the DSON spec as well as see it in action in an actual .duf file for themselves and protect themselves against false statements like the one you made above.

Except that is not the root cause of the problem. The biggest slowdown seems to be caused by Daz needing to create every slider in shape and posing pane with their associated erc formulas. It observation come from looking at filesystem activity when Daz load a figure and looking at what it actually read. 

The slider is a desired feature because it is beginner-friendly and many people want it. This is inherently an O(n) process. Daz needs to go through every file to get the sliders. Maybe Daz can optimise the process to be more efficient, or making it multi-threaded. But multi-threading would only buy you 2-5x performance improvement, considering most CPUs only have 4-8 cores. People have suggested only parsing the file to create dummy sliders and formulas and create the actual formulas as needed. But it is not clear to me that not creating the actual formula would be that much faster since the file still needs to be parsed completely.

Seems to me that the only solution is to go outside of the box and change the requirements. What Daz really needs is a way for the user to specify that it should load all morphs from a list of folders, all morphs from a list of product, and allow the user selectively loading morphs from the rest later.

It seems like there are many, many things that could be done, just none of them are being done.

Lets discuss what are those "many, many things" and which option you would prefer. Although given the dev's stated priority in the DS5 thread being getting the program out as soon as possible, I wouldn't hold hope for change until DS5 is released and bug fixed.

I was referring to you yourself, and the suggestions that Richard Haseltine suggested. But my argument is not that I know better than the devs, or that I can have a precise opinion on the subject of code that I have never seen. My position is primarily that it is not reasonable to think that the way things are now is simply the best they can possibly be, rather than the result of an architecture that was sufficient under older circumstances that is not sufficient now, because complex systems always present more opportunities for tradeoffs than professionals will have time to analyze and implement.

I agree with this, with the proviso that the other possibility is that a sufficient solution cannot be found that meets all the requirements for Daz at the current moment. It could be either the devs have not looked at this problem, or looked at it and can't think of anything better. I certainly do think it is more likely due to the lack of resources dedicated to solving this, but cannot rule out the possibility that an alternative solution does not exist that can meet all the requirements. Solution to the problem for a particular user is certainly possible, but I am not sure about the existence of a general solution. After all, it is much more likely that a solution for a problem does not exist than for a solution to exist. 

If someone could show what an alternative solution might look like, then that would then rule out the 2nd possibility and I would hope the dev would implement it. It would be more useful to discuss what a hypothetical solution is or if anything can be done to help people suffering the problem now, so I will leave it at that.

Leana

mindsong said:

I propose that a parallel issue being discussed here is the audacity of savvy programmer/users that would, without seeing the code (!), speculate as to why this is occurring, and even more irritating, folks that would propose reasons as to why it's still happening after all of these years and suggest (!) possible solutions.

The big difference being that they're not suggesting possible solutions, just saying "what Daz has done is bad" and that's all. Which benefits no one.

mindsong

Leana said:

mindsong said:

I propose that a parallel issue being discussed here is the audacity of savvy programmer/users that would, without seeing the code (!), speculate as to why this is occurring, and even more irritating, folks that would propose reasons as to why it's still happening after all of these years and suggest (!) possible solutions.

The big difference being that they're not suggesting possible solutions, just saying "what Daz has done is bad" and that's all. Which benefits no one.

edit:

.

--ms

TheMysteryIsThePoint

Leana said:

mindsong said:

I propose that a parallel issue being discussed here is the audacity of savvy programmer/users that would, without seeing the code (!), speculate as to why this is occurring, and even more irritating, folks that would propose reasons as to why it's still happening after all of these years and suggest (!) possible solutions.

The big difference being that they're not suggesting possible solutions, just saying "what Daz has done is bad" and that's all. Which benefits no one.

Your comment would only make sense in an Open Source context where an app's users have the right and I daresay the responsibility to maintain the app. There's a job for everyone.

There is no one here who could possibly, without having seen the code at all, make anything other than an officious suggestion to a person who has, when it's that person's job to maintain that code. Pointing out the things that make the user experience worse is all one can do. So that's what is done.

I don't know if it was your intention, but you did pretty succinctly highlight why you should prefer Open Source software.

 

Saxa -- SD

TheMysteryIsThePoint said:

I don't know if it was your intention, but you did pretty succinctly highlight why you should prefer Open Source software.

No she did not.  DAZ development, right or wrong, is based on suggestions that are considered.  That's not open-source per se.  You'll never know probably.   This is DAZ's modus operandi.   I have written my view long ago and it was clear that an issue tracker that was voted on was not on the list, or variations thereof.  If you and mindsong have issues, you are probably best to write a support ticket to DAZ.  Not that you can't keep posting away here.  Personally think DAZ team is more impressed long-run with those who endure and make stuff.  Despite any wishes to contrary.  But forums are about free-choice to a point.  Enjoy.

 

TheMysteryIsThePoint

Saxa -- SD said:

TheMysteryIsThePoint said:

I don't know if it was your intention, but you did pretty succinctly highlight why you should prefer Open Source software.

No she did not.

Yes, she did. She implied that comments about the poor user experience are more valid when coupled with a suggested remedy. How could a user who has not seen the code, conceivably give a useful remedy for a complex problem? If you'd like to educate yourself on how this occurs in practice, try reading the Blender Developer Meeting Notes and tell me if you think a Blender user could have contributed anything to the discussion that the devs didn't already know. The idea that a user can "help" the devs with suggestions having to do with technical aspects is only viable on an Open Source project because the entire user base has the same access to information that the devs do.

DAZ development, right or wrong, is based on suggestions that are considered.  That's not open-source per se.  You'll never know probably.

Just as you will probably never know on what Daz development is based. You refuted your own assertion much better than you refuted mine.

  This is DAZ's modus operandi.   I have written my view long ago and it was clear that an issue tracker that was voted on was not on the list, or variations thereof.  If you and mindsong have issues, you are probably best to write a support ticket to DAZ.

This is what infuriates me the most. The assumption that I am simply some immature, narcissistic, or asocial individual who has not already tried all of the possible means much more likely to actually affect change, including "write a support ticket". There is a reason I am triggered so easily and it is that I have never been treated so poorly by a company's support staff.

  Not that you can't keep posting away here.  Personally think DAZ team is more impressed long-run with those who endure and make stuff.

No, scratch the above, this is what infuriates me the most. By "make stuff", would having written an Alembic exporter that, unlike the official one, actually works and is supported despite my having a more-than-full-time job and a family, and giving it away and supporting it for free just because helping to enable people to do what they want to do is the most rewarding feeling I can describe, qualify? I'm not sure in what form the results of the Daz team's "being impressed" would manifest itself, or why they would have to be impressed in the first place, but at no time did any of the personnel to whom I made polite, humble, and directed requests for help respond at all. Not even a courtesy response. Long ago I was convinced that someone at Daz might think "Hey, this guy appears to have both the ability and motivation to contribute useful things. Maybe... we should help him?" So you might forgive me when I say that I have no idea what you are even talking about.

Beside that, requiring people to "impress" is without a doubt the worst support policy I can think of. And requiring them to "endure" is sadistic. How does either actually help Daz Studio evolve? Why even have an SDK at all, then?

I'll ignore the other sanctimonious parts.

mindsong

This is what infuriates me the most. The assumption that I am simply some immature, narcissistic, or asocial individual who has not already tried all of the possible means much more likely to actually affect change, including "write a support ticket". There is a reason I am triggered so easily and it is that I have never been treated so poorly by a company's support staff.

heh, you're a gem.

somewhere at the top of this thread is mention of some g8 laggy loading issue due to lotsa morphs or something like that. Looks a lot like the g1 with lotsa morphs laggy loading problem, and probably won't be fixed (yes, that's cynical speculation based on 10 years of observation. anybody is free to counter with evidence to contrary, and DAZ is free to prove me wrong - I could only hope).

the rest of this thread... confirms your quote above. it's absolutely gorgeous.

have a great day, and I recon it'd be better for our blood pressure to work on code rather than forums. hence my last 'edit'.

best,

--ms

mrinal

TheMysteryIsThePoint said:

Saxa -- SD said:

TheMysteryIsThePoint said:

I don't know if it was your intention, but you did pretty succinctly highlight why you should prefer Open Source software.

No she did not.

Yes, she did. She implied that comments about the poor user experience are more valid when coupled with a suggested remedy. How could a user who has not seen the code, conceivably give a useful remedy for a complex problem? If you'd like to educate yourself on how this occurs in practice, try reading the Blender Developer Meeting Notes and tell me if you think a Blender user could have contributed anything to the discussion that the devs didn't already know. The idea that a user can "help" the devs with suggestions having to do with technical aspects is only viable on an Open Source project because the entire user base has the same access to information that the devs do.

While I see where you guys are coming from, but being closed source doesn't mean that we are lacking in options to even approach the problem. I have not seen any attempts from Daz in remedying the problem let alone encourage user inputs in this case. Every time such conversations crop up the focus immediately shift towards damage control measures by suppressing further debate for the lack of insights or lack of better examples from this industry in mitigating this problem.

That said, several companies have issued bounties where they have openly solicited inputs from the community by carefully articulating the constraints without releasing their IP. The other more private approach is through floating RFPs and exchanging information through more more formal processes such as RFIs so that interested parties can submit their responses. But thats assuming Daz doesn't have the internal capacity or bandwidth to address the problems or find a solution from their end. Based on the fact that users have been complaining since the late Genesis 3 era, it doesn't look like a prioritization issue to me.

PerttiA

It isn't much use to vent ones frustrations here in the forums, DAZ in general doesn't follow or take part in these conversations.

Bejaymac

PerttiA said:

It isn't much use to vent ones frustrations here in the forums, DAZ in general doesn't follow or take part in these conversations.

Not to mention that Richard is a past master at dropping the axe on people, he's had plenty of practice doing it to me 

Leana

TheMysteryIsThePoint said:

Saxa -- SD said:

TheMysteryIsThePoint said:

I don't know if it was your intention, but you did pretty succinctly highlight why you should prefer Open Source software.

No she did not.

Yes, she did. She implied that comments about the poor user experience are more valid when coupled with a suggested remedy.

I answered a comment saying "how dare we criticize people for making suggestions" by pointing out that people were not, in fact, making suggestions. That has nothing to do with whether or not DS is open source.

And for the record, it is definitely possible to make suggestions on possible ways to modify a process without having seen the current code. You might end up suggesting things which are already done, but even that can be useful.

tonyjas46_1cccd9144e

With Genesis 9 having both male and female characters in the same content directory the loading times are now doubled if you have a lot of content. Clothing, matrerials poses are only referenced in the content directory BUT MORPHS are all loaded as soon as you select a figure. Isn't it time DAZ studio only loaded the essential base morphs for a figure and then selectively load all the required character morphs for that particular folder?

With V4 we had a list of optional morphs appear when V4 was selected to load. Surely Daz can use the same principle with later genesis versions. Seems to make a lot of sense when DAZ makes its income from content and not the program. I am reluctant to keep buying more characters when each set of morphs gets added to the existing list. I tend now to modify the base figure to create my characters for this reason.

Currently the only workaround for this problem is the have two morph directories and name one Morphs and the other Morphs zzz (or similar) then copy as needed from one to the other and delete or move to the Morphs zzz when not needed. An other option is to find all the sub folders under the Morphs folder and add "zzz" to the end of each one not used or needed. All in all this is a slow and time consuming process.

LenioTG

tonyjas46_1cccd9144e said:

With Genesis 9 having both male and female characters in the same content directory the loading times are now doubled if you have a lot of content. Clothing, matrerials poses are only referenced in the content directory BUT MORPHS are all loaded as soon as you select a figure. Isn't it time DAZ studio only loaded the essential base morphs for a figure and then selectively load all the required character morphs for that particular folder?

With V4 we had a list of optional morphs appear when V4 was selected to load. Surely Daz can use the same principle with later genesis versions. Seems to make a lot of sense when DAZ makes its income from content and not the program. I am reluctant to keep buying more characters when each set of morphs gets added to the existing list. I tend now to modify the base figure to create my characters for this reason.

Currently the only workaround for this problem is the have two morph directories and name one Morphs and the other Morphs zzz (or similar) then copy as needed from one to the other and delete or move to the Morphs zzz when not needed. An other option is to find all the sub folders under the Morphs folder and add "zzz" to the end of each one not used or needed. All in all this is a slow and time consuming process.

Hi, I'm still using G8, but I've basically solved the issue with this: https://www.daz3d.com/turbo-loader-for-genesis-8-and-81

Maybe RiverSoftArt will make a version for G9. It's a must for creators who have a ton of content (like me). I do comics, so once I've created the characters and their variations I don't need all the morphs, with this tool I let Daz know what morphs I'm using right now, and everything loads so much faster. If you want to turn on or off every morph, there's a way too.

Richard Haseltine

Turbo Loader is essentially a version of the fourth generation power loader (it doesn't work in exactly the same way, but both are enabling/disabling products or files). Daz Studio doesn't load the morphs, it loads their descriptions and links - otherwise it wouldn't be able to offer a slider or, once present, know when it should be triggered by other properties.

davorama_9f606bd07d

As a content creator I have to do a lot posing on the base G8 characters. Pose, reset, pose, reset, pose, reset.  It takes forever to reset the figure, just like loading a figure, as mentioned in this thread, if I have a lot of packages I've purchased for characters. The more custom morphs, the longer it takes to load and reset a character.  I have found that placing all my character packages for G8F in it's own library, (same with G8M) and unloading that library when I want to do pose sets will significantly improve the loading and reseting of the base G8 characters.  I do wish that DS was in this way, similar to poser, in that I can purchase a character package and apply it to the base V4 and save it back to the library, this custom character has only the morphs I want and not all of the morphs for every character package I've purchased.  It's a love/hate relationship, so the installing of characters to their own folder is my way of fixing the issue.

LenioTG

davorama_9f606bd07d said:

As a content creator I have to do a lot posing on the base G8 characters. Pose, reset, pose, reset, pose, reset.  It takes forever to reset the figure, just like loading a figure, as mentioned in this thread, if I have a lot of packages I've purchased for characters. The more custom morphs, the longer it takes to load and reset a character.  I have found that placing all my character packages for G8F in it's own library, (same with G8M) and unloading that library when I want to do pose sets will significantly improve the loading and reseting of the base G8 characters.  I do wish that DS was in this way, similar to poser, in that I can purchase a character package and apply it to the base V4 and save it back to the library, this custom character has only the morphs I want and not all of the morphs for every character package I've purchased.  It's a love/hate relationship, so the installing of characters to their own folder is my way of fixing the issue.

The solution for that is this pose set: https://www.deviantart.com/spyrorue/art/G8F-Reset-Poses-700458001

Don't ask me why, but it puts the figure in the base position resetting all poses in a fraction of a second, compared to the minutes it takes to actually reset with Daz Studio.

crosswind

Poser uses injection while Daz uses pre-load, each way has the pros and cons of its own... But the root reason of long loading time still results from the great deal of DSON support files under those well-known data folders. In fact, lots of people are  'character & morph collectors' who are keen on installing character products and varieties of morph / shape packages even if many of the 'products' are garbage. Moreover many of them don't understand the 'side-effect' of doing so, let alone the killer - 'duplicate formulas'....
BTW, 'thanks to' those 'vendors' esp. on some 3rd-party sites... Customers pay your bills, you steal their time  just because you're no pro or ignorant of these matters... or push your luck from time to time... Anyway, this is the real life in 3D world, but people still need to resist the impulse and make choices wisely...

crosswind

LenioTG said:

davorama_9f606bd07d said:

As a content creator I have to do a lot posing on the base G8 characters. Pose, reset, pose, reset, pose, reset.  It takes forever to reset the figure, just like loading a figure, as mentioned in this thread, if I have a lot of packages I've purchased for characters. The more custom morphs, the longer it takes to load and reset a character.  I have found that placing all my character packages for G8F in it's own library, (same with G8M) and unloading that library when I want to do pose sets will significantly improve the loading and reseting of the base G8 characters.  I do wish that DS was in this way, similar to poser, in that I can purchase a character package and apply it to the base V4 and save it back to the library, this custom character has only the morphs I want and not all of the morphs for every character package I've purchased.  It's a love/hate relationship, so the installing of characters to their own folder is my way of fixing the issue.

The solution for that is this pose set: https://www.deviantart.com/spyrorue/art/G8F-Reset-Poses-700458001

Don't ask me why, but it puts the figure in the base position resetting all poses in a fraction of a second, compared to the minutes it takes to actually reset with Daz Studio.

I also frequently use these scripts but they don't reset non Daz official pose controls and expression single / control dials... they just set 0 to bones / ectrl / pctrl dials that come from Base figure, so that's why it's faster than using Zero Figure Pose...

Richard Haseltine

crosswind said:

Poser uses injection while Daz uses pre-load, each way has the pros and cons of its own... But the root reason of long loading time still results from the great deal of DSON support files under those well-known data folders. In fact, lots of people are  'character & morph collectors' who are keen on installing character products and varieties of morph / shape packages even if many of the 'products' are garbage. Moreover many of them don't understand the 'side-effect' of doing so, let alone the killer - 'duplicate formulas'....
BTW, 'thanks to' those 'vendors' esp. on some 3rd-party sites... Customers pay your bills, you steal their time  just because you're no pro or ignorant of these matters... or push your luck from time to time... Anyway, this is the real life in 3D world, but people still need to resist the impulse and make choices wisely...

Injection still requires channels for the morph to be injected into, and grouping data - that is what soem of the ExP files used by the foruth generation figures are, while the older figures had to have special CR2 (figure) files with the channels defined (which made mixing morph sets a pain). .pmd files can be injected into a figure without needing pre-defined channels, but those had a big memory impact initially.