My DS Categories have some duplicates. How can I fix that?
barbult
Posts: 24,259
Somehow I got some duplicate categories under the Default category. I have two Materials, Wardrobe, and Utilities subcategories. How can I clean that up? It makes a cluttered mess of Smart Content views. If I select a file and do Categorize, when I check one of the duplicate categories, it automatically checks the duplicate one, too. If I look at the duplicate categories, they seem to contain the same items. It is as if I have two pointers to the same thing. Please help me fix this.
The details:
I'm not sure how these duplicates got created, but I think it happened yesterday when I accidently imported old User Data. That caused a huge number of log warnings/errors about files it could not find. I decided the easiest way to recover was to restore the whole CMS folder from the automatic daily backup that was done earlier in the day. I closed DS and DIM and checked Task Manager to be sure they were gone. PostgreSQL was also no longer running. Then I replaced the CMS folder with the backup. When I restarted DS, many files seemed to be uncategorized. In retrospect, I think DS/CMS was busy updating things in the background. As time progressed, the categories on items reappeared. However, because I saw missing categorization, I tried to use Content DB Editor to do some categorization of a product. That effort didn't "stick". Although I clicked Accept, when I went back in, the categorization was not there. Maybe that effort caused the duplicates????
Anyway, I just want to get rid of the duplicates. Can you help?
Screenshots are attached, but sometimes the forum hides images in the first post of a new thread.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Not really seen that before but from the looks of it I would guess that you have somehow ended up with two entries in the Categories table with the same name and parent id - even tho that shouldn't really be possible given that there is a unique index on those two and from what I can see there is also some sort of unique constraint. Although I am much more familiar with MySQL rather than PostreSQL so probably missing something there. That may explain the different behaviour whereby the Categorise window is using the names to control the checkboxes (meaning clicking one also clicks the clone) but the product view shows them as separate and only shows the asset in the "correct one".
I think the only way to know for sure is to dig into the table with an SQL editor which is maybe not something to do if you are not familiar with that kinda stuff.
If no-one offers a better solution, then I would recommend exporting your UserData changes, resetting the database and re-importing via the CMS Database tools. That SHOULD remove the clones if they are legitimate duplicates.
However, that won't work if those clones are not actually duplicates but instead categories with odd/dodgy/invisible characters which are being parsed/filtered differently by the different parts of the Daz interface. However, I think this scenario is less likely than the other to have happened without anything deliberate.
Thanks for responding, SofaCitizen. I know nothing about SQL, so I think I'd better not mess with CMS database tables in a SQL editor.
I did consider resetting the database. but I'm hoping someone like Richard Haseltine knows of a less drastic remedy. If not, I may have to go that way.
Yeah, hopefully Richard will have seen this before and know of something better. If not that is at least a plan Z if nothing else works.
Well, perhaps one thing you can do without resorting to digging around in the database is use a script to list the categories of one of the assets that is tagged in one of the cloned categories. There is a script on the wiki that can do this. For this script you would just replace the line at the bottom ("People/Genesis 3 Female/Characters/Victoria 7.duf") with the relative path of one of the asset items that is categoried in one of the clones. To be more helpful in this instance, I would recommend making a small change to the last print line to the below:
This will just wrap the Category paths in single quotes so that you can see if any of those contain a space at the beginning or end.
You can paste the code into the "Script IDE" panel (add this if you on't have it) and click the execute button which will display the results underneath as per the attached example.
It will be interesting to see if both clones are listed in the output. This does not list the category IDs tho unfortunately.
That sounds helpful, thanks. I'll try it when I'm back at my computer. Your instructions look clear. I think I'll be able to do it. I'm sort of familiar with the script IDE.
Hopefully that provides a bit more useful info.
Just as an addition that I have just noticed. The script seems to be case-sensitive which is a bit annoying since if you select an asset in Daz's product viewer it lists the physical path in the info tab underneath - but for some strange reason this is forced to lowercase. So you either have to type it in manually or paste it and then re-capitalise it to match the actual folder/file names as per the example attached.
@barbult, my first thought would be to delete the file from the Content Library, uninstall, then reinstall with DIM, and if you had a zip file stored, refresh the zip file's data, and update metadata within DIM before opening D|S.
I haven't done this particular error, but I have got close with two files loaded. One by DIM, one by Content Manager. Why I never login now.
Mary
It's not single product, it is 3 entire categories, with thousands of files in those categories.
Thanks for the extra clue. I'm back home now and I am going to give the script a try.
I ended up creating separate copies of the script and editing each one to specify a file in the duplicated categories. I didn't use the IDE. The results were printed to the log file. First there is the original with Victoria 7, where the only change I made was the suggested change to the print statement. After that there is a Wardrobe item, then a Materials item, the a Utilities item. None of the script results show any indication of duplicate categories. Here are the results
2023-03-06 19:39:12.099 [DEBUG] :: ----- Victoria 7 -----
2023-03-06 19:39:12.156 [DEBUG] :: Name: Victoria 7
2023-03-06 19:39:12.157 [DEBUG] :: Categories:
2023-03-06 19:39:12.160 [DEBUG] :: - '/Default/Figures/People/Female/Real World'
2023-03-06 19:39:12.161 [INFO] :: Script executed successfully: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories BV modified.dsa
2023-03-06 19:41:54.732 [INFO] :: Loading script: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Bubble Dress.dsa
2023-03-06 19:41:54.735 [DEBUG] :: ----- dForce Boho Boutique Bubble Dress -----
2023-03-06 19:41:54.738 [DEBUG] :: Name: Boho Layers Bubble Dress
2023-03-06 19:41:54.738 [DEBUG] :: Categories:
2023-03-06 19:41:54.740 [DEBUG] :: - '/Default/Wardrobe/Dresses'
2023-03-06 19:41:54.740 [INFO] :: Script executed successfully: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Bubble Dress.dsa
2023-03-06 19:45:20.854 [INFO] :: Loading script: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Bubble Dress Material Jacket.dsa
2023-03-06 19:45:20.857 [DEBUG] :: ----- dForce Boho Boutique Bubble Dress -----
2023-03-06 19:45:20.859 [DEBUG] :: Name: BD 02 Jacket
2023-03-06 19:45:20.859 [DEBUG] :: Categories:
2023-03-06 19:45:20.861 [DEBUG] :: - '/Default/Materials/Iray/Wardrobe'
2023-03-06 19:45:20.862 [INFO] :: Script executed successfully: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Bubble Dress Material Jacket.dsa
2023-03-06 19:49:10.104 [INFO] :: Loading script: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Tiling Utiulity.dsa
2023-03-06 19:49:10.107 [DEBUG] :: ----- UHD Fuzzy Cloth - Shaders and Merchant Resource -----
2023-03-06 19:49:10.119 [DEBUG] :: Name: DTHDC - Tiling Preset 01
2023-03-06 19:49:10.119 [DEBUG] :: Categories:
2023-03-06 19:49:10.121 [DEBUG] :: - '/Default/Utilities/Fabric Settings'
2023-03-06 19:49:10.121 [INFO] :: Script executed successfully: N:/DAZ 3D/Studio/My DIM Library/Scripts/Daz Script Samples/DB_Asset_Categories Tiling Utiulity.dsa
OK, so the assets are only tagged with the one category and there does not appear to be any rogue spaces shown. I don't know if there is any more scripts or API calls that will allow a bit more of a dig into the DB config that we can use. I will try and look for some after work.
As an aside, I figured out how to work around the case bug, which was quite obvious in hindsight. A bit late now tho.
Ahh, OK, the changes I found this morning would only have saved a little bit of typing but still had the same result.
I did have a quick look over my lunchbreak to put together something to list info about existing categories but not been able to figure it out. I know it can be done as I have a few scripts by others than can do this and so it's pretty annoying that I cannot figure it out. And I can't even find any help on how to do it - which makes me think it's really easy and I am just too dumb to figure it out :( I'll have another go a bit later to see if my luck changes.
OK, I am making some progress even if it's a little ugly.
This script is a little complicated because it's a hacked-about version of this script from the wiki and the changes suggested in this forum comment and then with some changes to just output the items that we are interested in.
If you run it then it should attempt to just list the categories that are potentially cloned - see example output also attached. I am hoping that this will show 6 entries for you which will indicate that the clones are there and, like with the previous script will hopefully show if there are any trailing/leading spaces. There is a "verbose" option that will probably show way more than you need but might be useful if the main run is not helpful. To output the verbose option just replace false with true in the last line.
This is what I get, but I don't know where to go from here.
2023-03-07 15:49:12.432 [DEBUG] :: ----- Listing matching results from 21903 total categories found -----
2023-03-07 15:49:12.434 [DEBUG] :: - '/Default/Materials'
2023-03-07 15:49:12.436 [DEBUG] :: - '/Default/Materials'
2023-03-07 15:49:12.441 [DEBUG] :: - '/Default/Utilities'
2023-03-07 15:49:12.441 [DEBUG] :: - '/Default/Utilities'
2023-03-07 15:49:12.441 [DEBUG] :: - '/Default/Wardrobe'
2023-03-07 15:49:12.441 [DEBUG] :: - '/Default/Wardrobe'
2023-03-07 15:49:12.472 [INFO] :: Script executed successfully: N:/DAZ 3D/Studio/My DIM Library/Scripts/From Forum/SofaCitizen/List Default WUM Categories.dsa
I'll try Verbose.
Well, yes, where we go from here is a difficult question. However, this does establish that you have proper duplicated categories which should not be possible under normal circumstances. I am not sure if we can surgically remove categories via scripting or any other means outside going to the DB directly.
Well, it is possible to delete categories from the Category section of the Content Library. However, that may prove to not work as intended since we have already seen one DazStudio window getting confused about the duplicates and treating them as one and the same. However, perhaps that is a plan we can place ahead of resetting the database and re-importing?
Just a quick check, can you right-click and rename any of the clones that do not contain assets?
Verbose output looks like it lists everything twice, like above, but with all the subcategories listed.
Thanks for your continued help.
Oops, the stupid forum upladed the same attachment agaIN.
In the Category view, both identical category names show the same assets. It is in Smart Content where only one of the categories shows sub categories. That seems inconsistent, but that is what happens. I cannot rename any subcategory under the Default category. Rename is grayed out. Should I be able to rename default categories? I suspect they are protected from renaming.
Not sure how much help I am being - it seems that for every answer we get another question appears :/
No, I don't think you can rename any of the default categories. I imagine you can only do that with the ones that have a "u" overlay because those only exist in the CMS whereas the others are created from the DSX files. Was worth asking just in case tho as renaming is less destructive than deleting.
We might be nearing the point where we have to revert to one of the drastic options since I am unsure if we can target the IDs of a specific category via scripting - make sure you export your UserData from the Content DB Maintenance first tho.
Yes, I'm not ready to go "scorched earth" with reseting the data base yet.
Well, hopefully another kind and more knowledgable soul will chip in with some ideas. As I can keep looking but I think this may require better scripting knowledge than I have :(
For the record, resetting and re-importing the database shouldn't necessarily be that bad - providing that you export your manual changes first. I used to do it quite a bit in order to sync my Windows machine with my Mac. The only thing it won't restore (AFAIK) is any custom Stores that you may have created.
Well, exporting the metadata first into a UserData.dsx file should mean that when you come to do the re-import it should show up as the first item in the list of products to be re-imported. So you shouldn't lose any manually-set categorisations. If you have created products manually purely via the Daz interface then I am not sure if those are saved. I use Content Wizard to install all my third-party stuff and that tool creates a DSX file which means the re-import includes that.
Yeah, I do think that because no-one else has chimed in that it is probably one of those errors that most people have not encountered and so there is not an obvious route to correct it directly :(
OK, here goes... I am exporting user data and resetting the database.
I have reset the data base and imported metadata, including my user data. The duplicate categories are gone.
My custom groups still exist, but no files are assigned to them anymore.
Smart Content will no longer sort by Order Date; everything is alphabetical.
Smart Content will no longer sort by Install Date.
Filter by Context doesn't work some things I have tested. I don't know why that is. The compatibility bases and compatibilities in metadata seem OK, but the Scene IDs are no longer set up for those items. I don't know how all that is stored/reloaded in metadata. I can only hope that this resolves itself.
A few things I tested have worked. No rhyme or reason that I can see.
I'll let it run for a while and see if it improves.
Nothing that I created myself (clothes in Marvelous Designer, for example) works with Filter By Context anymore. The Scene Identifications are not correct anymore. In many cases, the Scene ID shows the Compatibility Base as <None> and has the correct Compatibility Base in the Auto-Fit Base field instead. Maybe this is a bug in the metadata reimport process or the saving process for User Data. Anyway, it has become a big hassle to manually restore all that for everything I have created. Bundled with the problem that my custom groups no longer have any assets assigned to them, I cannot even tell what version of my creations were the final "favorite" versions. I fear that if I ever have to reset the DB and reimport metadata again, it will all be lost again.
(By the way, SofaCitizen, I do not put any of this trouble on your discussion of resetting the DB. Since nobody else offered help, it was the only option I had left.)