What are y'all using to monitor VRAM usage?

GatorGator Posts: 1,304

Running into a weird thing with Studio 4.21.0.5 - sometimes my system still renders with GPU, but it's really slow.  It will be like 1/10th the speed.  Usually shutting Daz down and firing it back up will cure it.  It happened a number of times today so I rebooted the system and again it's fast.  I suspect I'm getting very close to consuming all the VRAM but not sure... would like something to monitor that to help figure out what's going on.

Running Windows 11.

Comments

  • crosswindcrosswind Posts: 6,994

    Most likely your rendering fell back to CPU... Rocommend GPU-Z, just monitor GPU load and Memory used...

  • jjoynerjjoyner Posts: 617

    I second GPU-Z.  It's free and can be downloaded at https://www.techpowerup.com/gpuz .  I have a modest setup (16 GB RAM and RTX 2060 with 6 GB VRAM, Win 10 Pro) so I am constantly monitoring RAM and VRAM - expecially with more recent assets that are more demading of both.

  • GatorGator Posts: 1,304

    crosswind said:

    Most likely your rendering fell back to CPU... Rocommend GPU-Z, just monitor GPU load and Memory used...

    It weird because that definitely isn't it.  I've been keeping Task Manager open, and my CPU usage isn't jumping up.  Task manager is still showing the GPU usage.

    Slight update, first time it has occurred during batch renders overnight.  The renders average about an hour.  It did three, then just slowed right down.  I don't see anything reported in the Studio log, reports using video card and normal progress reporting iterations... but it's far slower.  Instead of 10K iterations in about an hour, only 2402 iterations after 4 hours and 15 minutes.

  • GatorGator Posts: 1,304
    edited January 11

    I'll try out GPU-Z, also the version of Daz I am using most likely must not be clearing out VRAM fully between scenes - this occurred during a batch job when I was sleeping.

    Post edited by Gator on
  • gramgram Posts: 19
    edited January 11

    Msi afterburner, as i use it for other tuning with the gpu as well.

    Post edited by gram on
  • franky85franky85 Posts: 133

    In the task manager's details section you can also add columns for Dedicated GPU Memory and Shared GPU memory to see which program is using how much VRAM.

  • MelissaGTMelissaGT Posts: 2,611

    It sounds like your VRAM isn't clearing. 

    What GPU are you using? Ever since moving to a 3090 I haven't had any issues with VRAM. I don't think I've seen Daz drop to the CPU once in the last two years. You can get them used now from folks that have moved on to 4090's...I'd recommend picking up a card with as large a VRAM pool as you can get. 

  • I use task manager and CPU-Z. I should download GPU-Z at some stage.

  • AgitatedRiotAgitatedRiot Posts: 4,437

    You don't know how much you would miss 16 GB VRAM. My 3090 is returning from an RMA, and I'm using an RTX 2070 with 8 GB.VRAM. Man, how limiting.

  • GatorGator Posts: 1,304

    MelissaGT said:

    It sounds like your VRAM isn't clearing. 

    What GPU are you using? Ever since moving to a 3090 I haven't had any issues with VRAM. I don't think I've seen Daz drop to the CPU once in the last two years. You can get them used now from folks that have moved on to 4090's...I'd recommend picking up a card with as large a VRAM pool as you can get. 

    I already have a 4090!  That's what's really annoying.

    I found the culprits(s).  One, I had three figures with 8K maps - only for the torsos and only bump and normal maps.  That was putting me on the ragged edge with 5 figures in a scene.

    The major culprit is that Daz Studio is not releasing all the VRAM used after using Iray in the preview render.  Here's a few screenshots after some optimizations.  In Iray preview I was using 18 GB IIRC, after switching the preview back to Texture Shaded my VRAM usage dropped to 8 GB.  Clearing the scene did basically nothing.  After closing down Daz Studio, VRAM usage dropped to 3.8 GB.

     

     

    GPU-Z Before Closing.jpg
    391 x 540 - 123K
    GPU-Z After Closing.jpg
    402 x 546 - 122K
  • PerttiAPerttiA Posts: 10,024

    Gator said:

    The major culprit is that Daz Studio is not releasing all the VRAM used after using Iray in the preview render.  Here's a few screenshots after some optimizations.  In Iray preview I was using 18 GB IIRC, after switching the preview back to Texture Shaded my VRAM usage dropped to 8 GB.  Clearing the scene did basically nothing.  After closing down Daz Studio, VRAM usage dropped to 3.8 GB.

    Makes sense, as when one does the full render, DS keeps the textures in VRAM for the next render of the same scene, and apparently it does the same for the preview but the catch is that the textures for the preview are probably not the same as for the full render, which results in two sets of textures being stored in VRAM.

    Remedy = Don't use Iray preview. I never use it as getting a good enough full render for checking, takes about a minute 

  • crosswindcrosswind Posts: 6,994
    edited January 12

    The root cause of OP's case should be the appr. 2.2GB VRAM consumed by IRay Render Engine. This part of consumption can only be released after DS is closed. Except for this part , the consumption of scene's geometry/textures can be 99% released after saving the render or closing Render window.  OP's case could be figured out as below: (with approximate values ~~ )

    - Before loading DS session, VRAM Used - 3.8GB
    - After loading DS and the scene, VRAM Used - 5.8GB (2GB delta VRAM consumption come from DS session and Scene itself)
    - After Ctrl + 0, VRAM Used - appr. 23.8GB (already very much approaching the critical point). This 18GB+ consumption consisted of 2.2GB Iray Engine consumption and 15.8GB geometry / texture consumption.
    - After Ctrl + 9, 15.8GB was released, but 2.2GB+ couldn't, now VRAM Used -  more likely >8GB.

    Then when clicking to final render, the availabe VRAM was actually not really enough (< 15.8GB) then it fell back to CPU render. If checking the log, there should be a message indicating "falls back to CPU...".

    Pls double-check.... It wouldn't make sense if it was not the case as above-mentioned, as with a 4090, rendering the same scene with GPU shouldn't have resulted in 10x render time...

    Post edited by crosswind on
  • GatorGator Posts: 1,304

    crosswind said:

    The root cause of OP's case should be the appr. 2.2GB VRAM consumed by IRay Render Engine. This part of consumption can only be released after DS is closed. Except for this part , the consumption of scene's geometry/textures can be 99% released after saving the render or closing Render window.  OP's case could be figured out as below: (with approximate values ~~ )

    - Before loading DS session, VRAM Used - 3.8GB
    - After loading DS and the scene, VRAM Used - 5.8GB (2GB delta VRAM consumption come from DS session and Scene itself)
    - After Ctrl + 0, VRAM Used - appr. 23.8GB (already very much approaching the critical point). This 18GB+ consumption consisted of 2.2GB Iray Engine consumption and 15.8GB geometry / texture consumption.
    - After Ctrl + 9, 15.8GB was released, but 2.2GB+ couldn't, now VRAM Used -  more likely >8GB.

    Then when clicking to final render, the availabe VRAM was actually not really enough (< 15.8GB) then it fell back to CPU render. If checking the log, there should be a message indicating "falls back to CPU...".

    Pls double-check.... It wouldn't make sense if it was not the case as above-mentioned, as with a 4090, rendering the same scene with GPU shouldn't have resulted in 10x render time...

    Yep, I'm positive.  It's happened a number of times, and the previous night when it happened it was still rendering I did double-check the log.  It was rendering with GPU but really slow.  I don't have that log now.

    I had this in the log for example:

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (NVIDIA GeForce RTX 4090)

  • GatorGator Posts: 1,304

    PerttiA said:

    Gator said:

    The major culprit is that Daz Studio is not releasing all the VRAM used after using Iray in the preview render.  Here's a few screenshots after some optimizations.  In Iray preview I was using 18 GB IIRC, after switching the preview back to Texture Shaded my VRAM usage dropped to 8 GB.  Clearing the scene did basically nothing.  After closing down Daz Studio, VRAM usage dropped to 3.8 GB.

    Makes sense, as when one does the full render, DS keeps the textures in VRAM for the next render of the same scene, and apparently it does the same for the preview but the catch is that the textures for the preview are probably not the same as for the full render, which results in two sets of textures being stored in VRAM.

    Remedy = Don't use Iray preview. I never use it as getting a good enough full render for checking, takes about a minute 

    Makes sense but it shouldn't happen... what's the use of caching textures and possibly geometry if you aren't going to use it?  laugh

  • PerttiAPerttiA Posts: 10,024

    Gator said:

    PerttiA said:

    Gator said:

    The major culprit is that Daz Studio is not releasing all the VRAM used after using Iray in the preview render.  Here's a few screenshots after some optimizations.  In Iray preview I was using 18 GB IIRC, after switching the preview back to Texture Shaded my VRAM usage dropped to 8 GB.  Clearing the scene did basically nothing.  After closing down Daz Studio, VRAM usage dropped to 3.8 GB.

    Makes sense, as when one does the full render, DS keeps the textures in VRAM for the next render of the same scene, and apparently it does the same for the preview but the catch is that the textures for the preview are probably not the same as for the full render, which results in two sets of textures being stored in VRAM.

    Remedy = Don't use Iray preview. I never use it as getting a good enough full render for checking, takes about a minute 

    Makes sense but it shouldn't happen... what's the use of caching textures and possibly geometry if you aren't going to use it?  laugh

    It does use them the next time one does a full render or Iray preview, but the two cache sets are not the same because preview uses lower quality.

    For full render one can test it by rendering the scene once (without activating the Iray preview), closing the finished render, and rendering the scene the second time, the maximum VRAM usage does not get any higher, even though after closing the first rendering window the usage of VRAM stays a lot higher than it was before starting the first render.

  • crosswindcrosswind Posts: 6,994
    edited January 12

    Gator said:

    crosswind said:

    The root cause of OP's case should be the appr. 2.2GB VRAM consumed by IRay Render Engine. This part of consumption can only be released after DS is closed. Except for this part , the consumption of scene's geometry/textures can be 99% released after saving the render or closing Render window.  OP's case could be figured out as below: (with approximate values ~~ )

    - Before loading DS session, VRAM Used - 3.8GB
    - After loading DS and the scene, VRAM Used - 5.8GB (2GB delta VRAM consumption come from DS session and Scene itself)
    - After Ctrl + 0, VRAM Used - appr. 23.8GB (already very much approaching the critical point). This 18GB+ consumption consisted of 2.2GB Iray Engine consumption and 15.8GB geometry / texture consumption.
    - After Ctrl + 9, 15.8GB was released, but 2.2GB+ couldn't, now VRAM Used -  more likely >8GB.

    Then when clicking to final render, the availabe VRAM was actually not really enough (< 15.8GB) then it fell back to CPU render. If checking the log, there should be a message indicating "falls back to CPU...".

    Pls double-check.... It wouldn't make sense if it was not the case as above-mentioned, as with a 4090, rendering the same scene with GPU shouldn't have resulted in 10x render time...

    Yep, I'm positive.  It's happened a number of times, and the previous night when it happened it was still rendering I did double-check the log.  It was rendering with GPU but really slow.  I don't have that log now.

    I had this in the log for example:

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (NVIDIA GeForce RTX 4090)

    Well, there might be another possibility... What's the Nvidia driver version that you use ?  If it's 546.01+, there's a new feature to prevent the rendering from "full falling back" to CPU in driver settings, which means a combo rendering with both GPU + CPU. So pls check if you have it activated, as well as checking if CPU has higher load when rendering. It is said that new feature may result in unexpected result....

    I uninstalled that 546 driver two weeks ago becuase it has a bug to work with the latest DS Public Build and I've never had a chance to test that new function since I have 48GB VRAM....

    Pls give it a check and further test.

    Post edited by crosswind on
  • GatorGator Posts: 1,304
    edited January 12

    crosswind said:

    Gator said:

    crosswind said:

    The root cause of OP's case should be the appr. 2.2GB VRAM consumed by IRay Render Engine. This part of consumption can only be released after DS is closed. Except for this part , the consumption of scene's geometry/textures can be 99% released after saving the render or closing Render window.  OP's case could be figured out as below: (with approximate values ~~ )

    - Before loading DS session, VRAM Used - 3.8GB
    - After loading DS and the scene, VRAM Used - 5.8GB (2GB delta VRAM consumption come from DS session and Scene itself)
    - After Ctrl + 0, VRAM Used - appr. 23.8GB (already very much approaching the critical point). This 18GB+ consumption consisted of 2.2GB Iray Engine consumption and 15.8GB geometry / texture consumption.
    - After Ctrl + 9, 15.8GB was released, but 2.2GB+ couldn't, now VRAM Used -  more likely >8GB.

    Then when clicking to final render, the availabe VRAM was actually not really enough (< 15.8GB) then it fell back to CPU render. If checking the log, there should be a message indicating "falls back to CPU...".

    Pls double-check.... It wouldn't make sense if it was not the case as above-mentioned, as with a 4090, rendering the same scene with GPU shouldn't have resulted in 10x render time...

    Yep, I'm positive.  It's happened a number of times, and the previous night when it happened it was still rendering I did double-check the log.  It was rendering with GPU but really slow.  I don't have that log now.

    I had this in the log for example:

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2024-01-11 21:57:31.564 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (NVIDIA GeForce RTX 4090)

    Well, there might be another possibility... What's the Nvidia driver version that you use ?  If it's 546.01+, there's a new feature to prevent the rendering from "full falling back" to CPU in driver settings, which means a combo rendering with both GPU + CPU. So pls check if you have it activated, as well as checking if CPU has higher load when rendering. It is said that new feature may result in unexpected result....

    I uninstalled that 546 driver two weeks ago becuase it has a bug to work with the latest DS Public Build and I've never had a chance to test that new function since I have 48GB VRAM....

    Pls give it a check and further test.

    I'm running 546.33.  This is the closest thing I see, fallback policy is set to Driver Default.

     

    GeForce_3D_Settings.jpg
    1010 x 696 - 181K
    Post edited by Gator on
Sign In or Register to comment.