Loading...
> >

Non-DIM scanner

TaozTaoz Posts: 9,940
edited March 2014 in The Commons

I've written a small freeware tool that will scan the order pages for items that are not downloadable via DIM, if anyone's interested (Windows only, sorry, but it should run on a MAC using CrossOver). It's currently in alpha, but very stable and no known bugs. Just a small fun project (I already had almost all the code from other projects anyway). :)

Currently not much error handling so it may crash if the server gets a hiccup or times out, but it hasn't happened to me more than a couple of times. Just kill it via Task Manager if it won't shut down, or if you want to abort the scanning for some reason. The GUI will "freeze" during scanning (like Poser during a render) as it's singlethreaded, but the page counter will keep updating so you can see if it's still running.

It's simple to use:

1. First go to the Tools menu and select "Set Browser Mode". If you don't, page formatting in the browser may be messed up. In XP it will be messed up as IE8 mode is no longer is compatible with the DAZ site, but the download pages are still usable for download. Not sure about IE9, but IE 10 and 11 should be OK.

2. Log in. Wait until you see the green "Logged in" next to the "Log In" button.

3. Click the "Scan Orders" button.

It will first scan the order list and then the order pages. Takes a little over 10 minutes per 500 orders for me, but it may vary depending on different external factors.

After the scan is finished, clicking on an item on the list will open the corresponding order page in the browser (you must log in manually if it's not logged in already (it will use the login cookie from IE if you use that for browsing DAZ)).

You can save the list via the Tools menu, and later reopen it without logging in.

Let me know if you run into any bugs etc..

Tested on XP and Win 7 / 64. Just unzip into a folder and run (may need to be run as Administrator, depending on the system) Don't install in any of the Program Files folders in Vista and later, as it will not run correctly from there (use the ProgramData folder for example).

Download here:

http://taosoft.dk/software/freeware/nds-daz/nds-daz-a0502.zip

--

nondimscanner.jpg
1220 x 709 - 250K
Post edited by Taoz on
«1

Comments

  • mrposermrposer Posts: 1,130
    edited December 1969

    Hey thanks.... very useful tool for finding products you need to install manually. Here is how my first run went.

    Windows 7 64bit... I have Chrome and Internet Explorer browsers.

    What worked:
    1. It processed all my 1276 pages of DAZ orders (yes I am a DAZ product horder) in a reasonable time .. I think about the speed you mentioned. I just passed time reading forum notes until it dinged when finished. It did not crash so far.
    2. The products listed that it found were indeed products that needed to be installed manually. I looked up several thru my Product library in another browser session to check. This is alot quicker than going order by order to find these products. Of course I have no idea if any products were missed that need to be installed manually but this works so far.

    What didn't work:
    1. The order number list generated was not correct... they were not even close to the correct orders.. not even the right number of digits.
    2. The DAZ page panel on the right had some formatting issues at the top but the order listed on that side of the page was correct.

    (I am reluctant to include screen shots of my order information but as an example it listed order # 9963961 for The Display Room which was incorrect order# but when I clicked on that item it brought up the correct order #101262333 on the right panel showing the DAZ Store info for that order)

    What I did was open another browser session for the DAZ Store and used my product library to search for each product in your list and verified the correct order number.

    Other suggestions/Requests
    1. Eliminate Platinum Club renewals from the list
    2. Have a pause button if we only want to check part of our order list
    3. I know you have a save for the list but a selection/copy/paste again would help to create a partial list to work on outside your utility

    Again thanks for building this utility!! It will be very useful... at least until DAZ finishes putting the full catalog on DIMM if that is even happening.

  • JonnyRayJonnyRay Posts: 1,744
    edited December 1969

    This is interesting. As someone with over 1600 DAZ products, it would be nice to know what it is that DIM didn't grab for me the last time I re-installed. Going to download it and take a look.

  • TaozTaoz Posts: 9,940
    edited March 2014

    MrPoser said:

    What didn't work:
    1. The order number list generated was not correct... they were not even close to the correct orders.. not even the right number of digits.


    You're right - there is a problem here. But not with all numbers though - 8573609 and lower are correct in my order history - they should be in yours too.

    The date for where it breaks relates to the date where the new Magento store was introduced (april 2012), after which the orders got two different sets of order numbers, one for Magento and one for the old database which still hasn't been phased out. This means that the two sets of numbers which both are listed in the HMTL code are identical up until a certain date after which they split, so there is one that matches the numbers on the order page, and another that only does partially. Stupid bug - I was aware of it but for some reason I got things mixed up anyway. Will be fixed ASAP.

    MrPoser said:

    2. The DAZ page panel on the right had some formatting issues at the top but the order listed on that side of the page was correct.


    OK - which version of IE are you using? I'm using IE 11 in Win 7 / 64 and all looks fine here (see screenshot).

    I assume you set the browser mode before you ran the program otherwise it will run in IE 7 mode which will make a lot of webpages break these days.


    Other suggestions/Requests
    1. Eliminate Platinum Club renewals from the list
    2. Have a pause button if we only want to check part of our order list
    3. I know you have a save for the list but a selection/copy/paste again would help to create a partial list to work on outside your utility

    I've thought about this and other things, but some of it requires multithreading which complicates things a bit. If I get the time I'll look into it.


    Again thanks for building this utility!! It will be very useful... at least until DAZ finishes putting the full catalog on DIMM if that is even happening.

    You're welcome, and thanks for the feedback...

    ie11-op-top.jpg
    986 x 292 - 105K
    Post edited by Taoz on
  • TaozTaoz Posts: 9,940
    edited December 1969

    MrPoser said:

    2. The products listed that it found were indeed products that needed to be installed manually. I looked up several thru my Product library in another browser session to check. This is alot quicker than going order by order to find these products. Of course I have no idea if any products were missed that need to be installed manually but this works so far.

    If the data on the order pages is consistent (which I suppose it is since it's generated dynamically) it shouldn't miss anything. Items with no DIM support have a different tag than those with so it should get them all.

    It is however possible that it may skip an item if there is a server timeout. This will be reported in the error.log file however so if there isn't any error message there it should be OK. I needs some improvement here so it notifies you directly, or whatever appropriate, if anything goes wrong.

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    Where are the error log and saved lists?

  • Lissa_xyzLissa_xyz Posts: 6,116
    edited March 2014

    Thanks for this. I have this scanning atm. Seems ok so far.

    fixmypcmike said:
    Where are the error log and saved lists?

    I have an error.log in the same folder as the app.

    /edit
    First run and it was already beneficial. I completely forgot I grabbed the Poser Format Exporter. lol

    Post edited by Lissa_xyz on
  • mrposermrposer Posts: 1,130
    edited December 1969

    savedlist will be in a data subfolder. It can be opened with notepad in windows

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    My mistake, forgot to extract from the zip before running it the first time.

    I see a problem with your methodology, however -- orders will still show items that have been returned if you don't return the whole order.

  • TaozTaoz Posts: 9,940
    edited March 2014

    fixmypcmike said:

    I see a problem with your methodology, however -- orders will still show items that have been returned if you don't return the whole order.

    Hm, yea, you're right. No way to detect which these are in the order page code, it seems. They can be filtered out though by making a compare with the json data on the Product Library pages. Luckily you only have to scan one page to get these and I already have the code to parse the data so that should be doable with not too much work.

    Post edited by Taoz on
  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    Taozen said:
    fixmypcmike said:

    I see a problem with your methodology, however -- orders will still show items that have been returned if you don't return the whole order.

    Hm, yea, you're right. No way to detect which these are in the order page code, it seems. They can be filtered out though by making a compare with the json data on the Product Library pages. Luckily you only have to scan one page to get these and I already have the code to parse the data so that should be doable with not too much work.

    Is there a reason you can't skip the order data and just process the Product Library pages directly?

  • TaozTaoz Posts: 9,940
    edited March 2014

    fixmypcmike said:
    Taozen said:
    fixmypcmike said:

    I see a problem with your methodology, however -- orders will still show items that have been returned if you don't return the whole order.

    Hm, yea, you're right. No way to detect which these are in the order page code, it seems. They can be filtered out though by making a compare with the json data on the Product Library pages. Luckily you only have to scan one page to get these and I already have the code to parse the data so that should be doable with not too much work.

    Is there a reason you can't skip the order data and just process the Product Library pages directly?

    That was what I originally wanted to do, but the product/download pages in the Product Library are ajax generated and I wasn't sure how to access the data I needed. I emailed DAZ_Jon who has been very helpful before with info about the store, but he hasn't replied. Maybe he's no more with DAZ, I don't know. I've been looking through some of the info he gave me earlier though, and I think I've found what I need there. It will probably require writing a lot of new code though so I'm not sure which way I'll go right now.

    Post edited by Taoz on
  • TaozTaoz Posts: 9,940
    edited March 2014

    fixmypcmike said:
    Taozen said:
    fixmypcmike said:

    I see a problem with your methodology, however -- orders will still show items that have been returned if you don't return the whole order.

    Hm, yea, you're right. No way to detect which these are in the order page code, it seems. They can be filtered out though by making a compare with the json data on the Product Library pages. Luckily you only have to scan one page to get these and I already have the code to parse the data so that should be doable with not too much work.

    Is there a reason you can't skip the order data and just process the Product Library pages directly?

    BTW, to clear up any confusion, there's no need to scan each product/download page to get the json data, they're all on the Product Library main page which takes a few seconds to scan, and compare takes a second, so it's not something that will slow things down.

    Post edited by Taoz on
  • TaozTaoz Posts: 9,940
    edited December 1969

    Updated to version 503.

    Fixed: some order numbers on the list were wrong
    Updated: saved list now include link number to order page ( https://www.daz3d.com/sales/order/view/order_id/[link number] )

    http://taosoft.dk/software/freeware/nds-daz/nds-daz-a0503.zip

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    I'd happily pay for a version that would look at my old install folder and find products that have subsequently been installed in DIM in my new install folder and help me get rid of duplicates...

    I'm also still hoping someone will write a utility to help create simple metadata for non-DAZ and older products so I can more easily add them to the Content Manager.

  • TaozTaoz Posts: 9,940
    edited December 1969

    zigraphix said:
    I'd happily pay for a version that would look at my old install folder and find products that have subsequently been installed in DIM in my new install folder and help me get rid of duplicates...

    Not sure precisely what you mean here?


    I'm also still hoping someone will write a utility to help create simple metadata for non-DAZ and older products so I can more easily add them to the Content Manager.

    I've never really gotten into exploring that metadata thing. Any info about it anywhere?

    I think BTW I saw D3D talk in some thread about perhaps making such a utility, but I'm not sure.

  • TaozTaoz Posts: 9,940
    edited December 1969

    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)

  • DAZ_JonDAZ_Jon Posts: 582
    edited March 2014

    Yours might still be good to use for some. The one in the Product Library is an all or nothing thing. So, basically, there are products where parts of it aren't converted to be used in Install Manager and still needs cleanup... but those parts will show as being ready for Install Manager even though parts aren't able to be installed. So, there is a gap where it is more like "Install Manager Limbo" where it is only partially converted... those we aren't planning on adding a filter for as the goal is to convert every product over to be 100% Install Manager or not supported at all (like older plugins), and those are still most likely getting missed by people who might be looking for them.

    Post edited by DAZ_Jon on
  • TaozTaoz Posts: 9,940
    edited December 1969

    DAZ_Jon said:
    Yours might still be good to use for some. The one in the Product Library is an all or nothing thing. So, basically, there are products where parts of it aren't converted to be used in Install Manager and still needs cleanup... but those parts will show as being ready for Install Manager even though parts aren't able to be installed.

    You're talking about bundles where not all the included items have been converted yet?


    So, there is a gap where it is more like "Install Manager Limbo" where it is only partially converted... those we aren't planning on adding a filter for as the goal is to convert every product over to be 100% Install Manager or not supported at all (like older plugins), and those are still most likely getting missed by people who might be looking for them.

    But when all DIM capable products have been converted (and from then on new ones will all be DIM-ready on release I assume), there won't be any non-DIM items left that can't be found using a "dim::false" search (or any DIM items that can't found in a "dim::true" search), right?

    The problem is that my util is currently scanning the order pages only, not the item pages in the product library, so it won't show the DIM status of items included in bundles anyway, only the status of the bundles themselves (which may not be correct, if I understand you correctly). Something I hadn't really considered, I must admit. Ideally it should scrape the product pages in the library instead to get all data correct (I can see you gave me the ajax link for that earlier, so that should be doable). The question is how long it will take before all items have been converted to DIM. There don't seem to be many left so maybe there isn't much point in doing this.

  • wizwiz Posts: 1,100
    edited December 1969

    Taozen said:
    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)


    No "maybe" there, my friend. I've used that same strategy with other companies, and I've seen other people do it many times. It's highly effective.

    Thank you for lighting a flame under them.

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    Taozen said:

    zigraphix said:

    I'm also still hoping someone will write a utility to help create simple metadata for non-DAZ and older products so I can more easily add them to the Content Manager.

    I've never really gotten into exploring that metadata thing. Any info about it anywhere?

    I think BTW I saw D3D talk in some thread about perhaps making such a utility, but I'm not sure.

    I don't think there's any official documentation, but it's XML, it's pretty easy to read. I don't know if D3D is working on it.

    But it could combine with the other store utilities to make a "DIM for non-DAZ Products," effectively....

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    Are you talking about creating metadata, or being able to install non-DAZ products with DIM? Two different things.

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    fixmypcmike said:
    Are you talking about creating metadata, or being able to install non-DAZ products with DIM? Two different things.

    Both, and I know they're two different things. :)

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    zigraphix said:
    fixmypcmike said:
    Are you talking about creating metadata, or being able to install non-DAZ products with DIM? Two different things.

    Both, and I know they're two different things. :)

    I figured you did, just wanted to make sure people reading the thread didn't get the wrong impression.

  • TaozTaoz Posts: 9,940
    edited December 1969

    wiz said:
    Taozen said:
    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)


    No "maybe" there, my friend. I've used that same strategy with other companies, and I've seen other people do it many times. It's highly effective.

    Thank you for lighting a flame under them.

    I also suspect that the old DAZ Content Downloader I wrote may have been playing a role in DAZ decision to create DIM. If that's the case I haven't lived in vain, I guess... :)

  • TaozTaoz Posts: 9,940
    edited December 1969

    zigraphix said:
    Taozen said:

    I've never really gotten into exploring that metadata thing. Any info about it anywhere?

    I think BTW I saw D3D talk in some thread about perhaps making such a utility, but I'm not sure.

    I don't think there's any official documentation, but it's XML, it's pretty easy to read. I don't know if D3D is working on it.

    But it could combine with the other store utilities to make a "DIM for non-DAZ Products," effectively....

    Where are these XML/metadata files located?

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    Taozen said:
    zigraphix said:
    Taozen said:

    I've never really gotten into exploring that metadata thing. Any info about it anywhere?

    I think BTW I saw D3D talk in some thread about perhaps making such a utility, but I'm not sure.

    I don't think there's any official documentation, but it's XML, it's pretty easy to read. I don't know if D3D is working on it.

    But it could combine with the other store utilities to make a "DIM for non-DAZ Products," effectively....

    Where are these XML/metadata files located?

    The CMS metadata is in Runtime:Support:*.dsx
    The DIM manifests are inside the product zip file named manifest.dsx

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    Taozen said:
    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)

    Wait, where is this in the product library? Here on the site? I must be going blind....

  • fixmypcmikefixmypcmike Posts: 19,583
    edited December 1969

    zigraphix said:
    Taozen said:
    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)

    Wait, where is this in the product library? Here on the site? I must be going blind....

    Type
    DIM::false
    in the filter box.

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    Taozen said:
    zigraphix said:
    I'd happily pay for a version that would look at my old install folder and find products that have subsequently been installed in DIM in my new install folder and help me get rid of duplicates...

    Not sure precisely what you mean here?


    I don't know if I'll be any more coherent tonight, but I'm waiting while the CMS rebuilds (again) so here goes...

    I had all my content downloaded and installed. Then DIM came out. I set DIM up and re-downloaded a lot of content to a new content folder. At that point I had a lot of content installed twice, so things would show up twice in the Smart Content folder, and I got a lot of "duplicate ID" errors. I tried some utilities to get rid of duplicate files between the two content directories, and realized this was a bad idea when I ended up deleting thumbnails for the DS versions of materials, for example, because they had the same name and file size as the Poser versions. Yuck. I stopped doing that in time to avoid completely mangling my old content directory, but now I still have some things installed twice, and some broken installs. In short, I need to reinstall a lot of stuff, and the installers are all mixed together on my backup drive-- I want to find just the installers that aren't in DIM, but that's easier said than done.

    So at this point I have:

    - a DIM content directory
    - a non-DAZ content directory (this was pretty easy, I was able to filter by just zips)
    - a bunch of DAZ installers I'm not sure whether or not to install again, because I don't know if they're in DIM or not (including some old freebies from the now-gone archive)

    So if I could just download the non-DIM installers, or restore an older content library that contains some content now in DIM and then weed out the stuff that's now in DIM (I didn't move files, so they should still be in their default locations), that would be very helpful.

    OMG, the CMS finally finished rebuilding, and doesn't seem to have crashed. O frabjous day, calloo callay.

    I don't know if what I typed made any sense, but I'm too tired tonight to try to figure it out further. If it's all gibberish, just say so and I'll try to explain it again tomorrow. Maybe there's nothing anyone can do about it anyway....

  • zigraphixzigraphix Posts: 2,787
    edited December 1969

    fixmypcmike said:
    zigraphix said:
    Taozen said:
    Well, I'll probably stop developing this thing further now that DAZ has added this feature to the product library.

    Maybe that's the way to make them add new features to the store - writing third party utilities that do the job... :)

    Wait, where is this in the product library? Here on the site? I must be going blind....

    Type
    DIM::false
    in the filter box.
    Which filter box? In the Product Library page here on the website? In DS? In DIM? (the first two don't seem to do anything interesting....)

Sign In or Register to comment.