Do DIM updates overwrite private tags and metadata?

gw3972

Before I embark on a long series of tagging products I'd like to know if DIM updates overwrite anything I've added myself to a product's metadata. I don't want to do a lot of work for nothing!

I can't find anything anywhere that mentions this.

Richard Haseltine

Make sure you export user data before applying the updates. Right-click on the Content Library pane's tab, or click its menu button, and select Content DB Maintenance, then check Export User Data and click Accept.

linvanchene

gw3972 said:

Before I embark on a long series of tagging products I'd like to know if DIM updates overwrite anything I've added myself to a product's metadata. I don't want to do a lot of work for nothing!

I can't find anything anywhere that mentions this.

Update / Edit
replaced the term "all metadata" with a more precise description:

Metadata that can have multiple values at the same time is not overwritten but added together by updates.

Examples: Tags, Categories

Still the state of the metadata is changed by installations. Categories that the user deleted before are installed again and need to be removed again to recreate the state before the update.

Metadata that can only be in one state will get overwritten by updates.

Examples: File Types

- - -

For users that only change "tags" and "categories" for normal day to day situations no backup may be needed if the user does not mind that other categories and tags may be added to his own by updates.

For those users it is also enough to export and import just the "User Data" when they want to install DS on a new system.

- - -

If you make very detailed changes to the products the more time consuming but safer way is to export the "User Product Data" .dsa and .dsx files from the Product Library "Content DB Editor" to /runtime/support and then make a backup in another location.

The benefit of doing it that way is that you have a backup of your metadata work for each single product and not just one file for all products.

The current function "Export User Data" only collects the CHANGES the user makes to the metadata. It is not a collection of the complete state of all items of a product.

Example:

The user changes the Category of an item I in a product from A to B.

The next product update by DAZ changes the category of all items I, II, III, IV, V included in the product from A to C.

By reimporting the "User Data" the user ends up having the combination IB, IIC, IIIC,IVC,VC.

But this is not what the user wanted. He wanted to keep the product in the state
IB, IIA, IIIA, IVA, VA.

The only way the user can save the complete state of the product is to export all product metadata information as .dsa and .dsx files from the "Product Library" "Content DB Editor".

- - -

To speed this process up I made a feature request to export ALL product data with the click of one button to /runtime/support

If you make a backup of that folder you could be sure that really all changes you ever made will be ready to be reinstalled later.

- - -

If you are interested in more details why just exporting the "User Data" might not be enough please have a look at the feature request report;:
1 click Batch export of all "Product" metadata from "Content DB Maintenance" "Export All Product Metadata"

https://bugs.daz3d.com/view.php?id=49185

- - -

You can find a tutorial about Adding Metadata to Products here:

http://wiki.daz3d.com/doku.php/public/software/dazstudio/4/userguide/creating_content/packaging/tutorials/adding_metadata/start

- - -

gw3972

Good grief, that looks complex! In other words, every time DIM tells me there is updated content, I will have to go through the export/download/re-import cycle?

It would be much better if DAZ's programmers had adopted a database schema that separated DAZ data from user data and made sure that DIM only updated DAZ data. (I learnt that "trick" from a very good DB analyst I once worked with. It saved a LOT of hassle all round.) Maybe I should file a feature request and see if it can be done retroactively.

Thanks for the links - I'll head off and read 'em now.

linvanchene

gw3972 said:

Good grief, that looks complex! In other words, every time DIM tells me there is updated content, I will have to go through the export/download/re-import cycle?

Update / Edit:
to be more precise:

If the only thing you care about is that your custom categories and tags remain then you may not need to go through the whole export / backup / import cycle.

If you want the exact state of the product as it was before the update you have to reimport a backup of the .dsx file.

Only this way you can prevent that additional categories that you do not want and deleted or unchecked before are installed again.
Only this way you can prevent that file types you changed are not overwritten.

- - -

Currently every improvement comes with the downside that I have to
- manually export and backup .dsa and .dsx files
- run the update from the DIM
- manually put my .dsa and .dsx files back into /runtime/support
- reimport the information stored in the .dsx files back into DS.

gw3972 said:

It would be much better if DAZ's programmers had adopted a database schema that separated DAZ data from user data and made sure that DIM only updated DAZ data.

There would be a solution for that issue:

"Completly" separate installers for the content and the metadata .dsa and .dsx files in /runtime support.

I stress the "completly" because in 2011 there were separte installers for metadata and content. Nevertheless as someone explained me for some 3rd party compatibility related reason the .dsa and .dsx files were also included in the main installers.

In 2012 the separate metadata installers were dropped because it became obvious that the 3rd party will not support the metadata system.

A more advanced solution would give the user the ability to choose each time when products are installed if the .dsa and .dsx files get replaced or not.

Maybe I should file a feature request and see if it can be done retroactively..

If you could file a feature request that would be incredibly helpful.

Currently it is only a small but steadily growing group of users that ask for more advanced ways to manage metadata.

From that point of view every new voice that asks for metadata support will help.

- - -

DAZ_Jon

That actually isn't correct according to the software developers. Any modified meta data will persist through an update of a product, it won't blow it out. You can export it out and save it as a backup if you want, but shouldn't be necessary.

RCDeschene

DAZ_Jon said:

That actually isn't correct according to the software developers. Any modified meta data will persist through an update of a product, it won't blow it out. You can export it out and save it as a backup if you want, but shouldn't be necessary.

I can confirm this! I've modified metadata on an item before, then some time later I saw the product it came from updated in my DIM and promptly installed the update. I return to my Studio and all was still as I made it.

You should be alright. : )

linvanchene

There is not just one type of metadata. There are several different types of situations that all work differently.

- - -

The short version of what I obverved is:

Metadata that can have multiple values at the same time is not overwritten but added together by updates.

Examples: Tags, Categories

This means
- the categories the user created should still be there
- categories that are included in the updated installer are added
- if the user had deleted some categories before they will be added again as well

All metadata that can only be in one state will get overwritten by updates.

Examples: File Types, "declared as" information,

- - -

The most important thing to understand about this is that the user may not want that categories he deleted or changed are added again by the update.

The user wants that the whole product stays in the very same state as it was before the DIM update.

Even if the categories the user created persist after a DIM update the addtional DAZ categories that are forced upon the user may be unwanted.

Example:

In 2011 the metadata used the categories /preset/materials/xy after Summer 2012 it was just /Materials/xy .

When installing updates that just fixed geometry but still has 2011 stye metaddata the old /presets materials are forced once again on the system even if the user took the time to completly adapt all metadata to the new official categories introduced 2012.

http://docs.daz3d.com/doku.php/public/dson_spec/format_description/metadata/categories/start

Additional work is created for the user because he has to delete the old "presets" categories again to clean up his installation.

RCDeschene said:

I can confirm this! I've modified metadata on an item before, then some time later I saw the product it came from updated in my DIM and promptly installed the update. I return to my Studio and all was still as I made it.

Can you remember what exactly you changed?

If you are just changing categories they will persist because they are also saved as "User Data" into the local databases.

But if you go check the categories you may notice that there are now not only your categories but there may be a tic in more than just one place. Chances are there are additonal checkmarks set in other categories than just the one you choose.

- - -

Metadata includes more than just "Categories". There are other cases like "file types" in which I observed a lot of times that they will just be overwritten.

DAZ_Jon said:

That actually isn't correct according to the software developers. Any modified meta data will persist through an update of a product, it won't blow it out. You can export it out and save it as a backup if you want, but shouldn't be necessary.

Can you be a bit more specific about this?

It would be very helpful if the software developers would find the time to have a look at what I write and then answer in detail what is true and where I am under a wrong impression.

At the moment I am under the impression that the metadata situation is not improving exactly because everyone believes that everything is working as it should. Those who understand how it works are tired to talk about it and the others who complain are told they are wrong because still some missunderstandings are happening because for many different situations the same term "metadata" is used.

It is not helpful to talk about "metadata" in general. One needs to look at each specific case and learn to understand how it works.

- - -

If it is true what you are saying I would have made a lot of unnecessary work for the past 12 months since I started creating a backup of /runtime/support each evening after I installed and modified content.

- - -

I observed many different cases:

- After installing a "DAZ Product Data" update the files in /runtime/support are getting overwritten.

So if a user did export .dsa and .dsx files to /runtime support all the data saved in /runtime/support is lost if he did not create a backup in another location.

Some of that data may still be present as "User Data" in one of the databases located in C:\ProgramData\DAZ 3D\Content Management Service\databases

But certainly not in its complete original state.
- Some data entries are added together
- Some data entries are overwritten

- - -

- Categories:

I noticed that after running a DIM update the user created categories may persist because they are also saved in the "User Data" part of the database

Instead of one tic in the categories there now are two or more tics in different places.

Nevertheless the user may not want that old categories like "Shoes" or " Props & Accessories" are written into the database over and over again whenever a new version of "Genesis Starter Essentials" is installed.

How many times do I have to manually update product metadata to the new categories

http://docs.daz3d.com/doku.php/public/dson_spec/format_description/metadata/categories/start

just to have my work remixed again with another update still featuring the old 2011 version of the categories just a week later?

- - -

- Tags:

May be the same as categories. Tags by the user and tags made by DAZ can both show up because they are not mutually exclusive

In general the more tags the better. Unless of course some tags just do not fit.

- - -

- File Types:

An item can only have one file type at a time.

As far as I have observed any changes I make to file types are overwritten by any DIM updates that are installed.

Example:

Enviroments: If I choose as file type "scene" and DAZ sets it as "Set" my choice gets overwritten
If I choose file type "prop" and DAZ sets it as "actor" my choice gets overwritten.

Another example

The "Visibility" file type is not helpful when used in combination with material presets

When the file type material is used the preset will show up in the surfaces tab.
When the file type visibility is used the preset will not show up in the surfaces tab.

I obvserved that after installing DIM updates my change to set the file type to "Materials" so that presets show up in the surfaces tab gets overwritten by the file type "Visibility" once again.

- - -

- Compatibility bases:

- It seems that Compatibility bases persist. After the V5 pro bundle was first updated with .duf file types some products now had double the amount of compatibility bases because someone deceided that it was a good idea to add some new ones for the .duf version.

I do not want double the amount of compatibility bases after I install a product update. That will just end up in confusion when assigning compatibilities to custom material presets..

- "Compatibilites" also add up to several versions that may or may not be wanted.

If the "Compatibilites" "declared as" that are set up in the Content DB Editor are replaced with new compatibility bases this breaks compatibility to texture product addons that still used the old compatibility bases.

- - -
- - -

I would be very greatful for a more detailed explanation which of my observations are not true.

To sum this all up:

I believe the basic missunderstanding that is happening is that most people just consider editing metadata as changing the categories and adding some tags. But that does not cover it all.

Some users are making more advanced changes like file types and those DO get overwritten by any DIM updates.

The other wrong assumption is that the user does not mind that some categories he deleted will show up again after updates.

- - -

If I remove a checkmark in a category A and place my own checkmark in category B and then the update just makes another checkmark in category A again I say:

The update did overwrite the complete state of my metaddata. Instead of being just in category B the item is now in categories A and B. And that really is not what I want.

A+B is not equal B.

If I unchecked a category I want it to remain unchecked even after product updates.

- - -

This is the reason why I say the only way I know to make sure that the complete state of a product remains after an update is to

- export the .dsa and .dsx files of the product I adjusted to /runtime/support
- make a backup of the whole folder in a different location
- install the product update with the DIM
- place a copy of the backup in /runtime/support
- reimport your custom version of the metadata from the .dsx file

- - -

Updated / Edit:

I updated and structured my post several times. Now it should be complete.
I hope I now provided enough examples that everyone can understand what I am talking about.

I really hope someone of the DAZ software developer team reads through it and can clear up some missunderstandings or confirm that I am not imagining all of this.

If you are a forum moderator please forward this post to the metadata team or those responsible for metadata software development.

- - -

I am really tired about all this metadata talk without seeing any improvments or confirmations.

The most frustrating part is when someone then just comes along and says

"It is not true" or "It works fine the way it is now" in a very general way without going into any further detail when I observe the opposite on a weekly basis.

linvanchene

I made a feature request to

0050585: Give the user an option to prevent the DIM from overwritting .dsa an .dsx files in /runtime/support

https://bugs.daz3d.com/view.php?id=50585

- - -

In the future request I asked that DAZ software developers also have a look at this thread and hopefully can provide more detailed information which of my observations are true and which are based on missunderstandings.

- - -

Again I understand that it may not be easy to come up with a solution.

An important first step is to make information available to the users, the forum and DAZ staff which processes are happening when product updates are installed by the DIM and what the detailed consequences are for metadata the user created themselves depending on the type of the data.

mark128

I have not tried creating my own metadata, but I have organized my own content categories in the content pane.

There is the content -> Default category. If you re-organize things in the Default category, it is likely to get messed up by future product downloads or updates of existing products.

I created a Content -> Mine category. I organize everything the way I like it under that category. DAZ installers will never overwrite or mess with information in the Mine category.

Now DAZ updates occasionally moves files around. When that happens the file my content entries are linked to disappears, and I need to update my content entries, but that does not happen very often.