CUDA doesn't have enough memory to launch the second time

duckdoom5duckdoom5 Posts: 11

So when using Iray to render my scene I get this weird issue that I've not seen many people talk about.

I had this problem before where it was very slow and I found out it was falling back on the CPU. I turned it off so I can now see when it tries to fallback.

So I can render everything fine once, but then when I try to start the next render session it seems like Iray / Daz didn't clear the CUDA launch kernel memory.

Restarting daz doesn't help. The only things that work are singing out/restarting my pc or reinstalling the graphics drivers. It's quite annoying to have to restart my pc every time I need to render for a second time.


Any ideas? Maybe someone knows a way to reset the memory?

Also, it doesn't appear to be VRAM. I have a bit less then 4 GB's still available (see attached ss), and if I add up the memory in the logs it's about 2.5 GB's

 

2021-09-23 12:22:36.772 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Initializing OptiX Prime for CUDA device 0
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info :     CUDA device 0 (NVIDIA GeForce GTX 1070)
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering...
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER ::   1.3   IRAY   rend progr: CUDA device 0 (NVIDIA GeForce GTX 1070): Processing scene...
2021-09-23 12:22:37.156 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : Geometry memory consumption: 1.003 GiB (device 0), 0.000 B (host)
2021-09-23 12:22:56.602 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : Texture memory consumption: 1.361 GiB for 237 bitmaps (device 0)
2021-09-23 12:22:56.616 Iray [INFO] - IRAY:RENDER ::   1.8   IRAY   rend info : Importing lights for motion time 0
2021-09-23 12:22:56.617 Iray [INFO] - IRAY:RENDER ::   1.8   IRAY   rend info : Initializing light hierarchy.
2021-09-23 12:22:56.778 Iray [INFO] - IRAY:RENDER ::   1.8   IRAY   rend info : Light hierarchy initialization took 0.161s
2021-09-23 12:22:56.779 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : Lights memory consumption: 1.365 MiB (device 0)
2021-09-23 12:22:56.906 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : Material measurement memory consumption: 0.000 B (GPU)
2021-09-23 12:22:56.986 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : PTX code (2.704 KiB) for SM 6.1 generated in 0.046s
2021-09-23 12:22:56.987 Iray [VERBOSE] - IRAY:RENDER ::   1.8   IRAY   rend stat : Materials memory consumption: 39.792 MiB (GPU)
2021-09-23 12:22:57.158 Iray [INFO] - IRAY:RENDER ::   1.8   IRAY   rend info : JIT-linking wavefront kernel in 0.073s
2021-09-23 12:22:57.191 Iray [INFO] - IRAY:RENDER ::   1.8   IRAY   rend info : JIT-linking mega kernel in 0.033s
2021-09-23 12:22:57.191 Iray [INFO] - IRAY:RENDER ::   1.3   IRAY   rend info : CUDA device 0 (NVIDIA GeForce GTX 1070): Scene processed in 20.414s

2021-09-23 12:22:57.192 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.3   IRAY   rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Not enough memory for kernel launches (53.762 MiB (304.762 MiB) required, 0.000 B available). Cannot allocate framebuffer.
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.3   IRAY   rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Failed to setup device frame buffer
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.3   IRAY   rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Device failed while rendering
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER ::   1.3   IRAY   rend warn : CUDA device 0 (NVIDIA GeForce GTX 1070) ran out of memory and is temporarily unavailable for rendering.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER ::   1.3   IRAY   rend warn : All available GPUs failed.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.3   IRAY   rend error: Fallback to CPU not allowed.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.3   IRAY   rend error: All workers failed: aborting render
2021-09-23 12:22:57.198 Iray Render error: Internal rendering error.
2021-09-23 12:22:57.237 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Device statistics:
2021-09-23 12:22:57.237 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (NVIDIA GeForce GTX 1070):      0 iterations, 20.421s init, 0.005s render

Post edited by duckdoom5 on

Comments

  • PerttiAPerttiA Posts: 4,503

    What are the specs of your computer and how much VRAM does the GTX 1070 have?

  • duckdoom5duckdoom5 Posts: 11
    edited September 2021

    I've attached a screenshot of my specs

    GPU Specs.png
    1090 x 816 - 58K
    CPU Specs.png
    670 x 265 - 25K
    Post edited by duckdoom5 on
  • duckdoom5duckdoom5 Posts: 11
    edited September 2021

    I don't know why, but it's not attaching my second ss.

    Edit: Got it attached, I guess it had issues with an & in the image file name

    Post edited by duckdoom5 on
  • PerttiAPerttiA Posts: 4,503

    Ok, what other programs do you have open and doing something at the same time?

    At which stage did yo take the screenshot of Task manager - 4.4GB's of 8GB's used at that point, not much left for DS.
    In an other screenshot, it's saying 29GB's of Page file used...

    You are also using W11, which can cause unseen glitches

  • duckdoom5duckdoom5 Posts: 11
    edited September 2021

    Thanks for trying to help. Windows 11 is definitely not the issue here as I had this before on windows 10. Also on less VRAM taxing scenes.

    Though after further investigation I found that when it actually is able to render the scene, it's using close to maximum amount of VRAM. (In other words, it's using way more memory then reported in the logs)

    I'm guessing that DAZ/ Iray doesn't release all the VRAM after a render, causing it to be insufficient when I render the second time.

    To answer your question, I took that ss of Task Manager right after I got the message it failed to start the render. I also only have daz open, no other programs

    Post edited by duckdoom5 on
  • PerttiAPerttiA Posts: 4,503

    duckdoom5 said:

    Though after further investigation I found that when it actually is able to render the scene, it's using close to maximum amount of VRAM. (In other words, it's using way more memory then reported in the logs)

    The OS, DS and the baseload of the scene are taking their share of the VRAM and using a GTX, means that some additional VRAM is used for emulating the RTX functions in software.

    With an RTX 2070 Super (8GB), According to the log, about the maximum the geometry, textures and working space can have is 5.7GB's for the scene still to render with the GPU - The GPU-Z reports total VRAM usage at this point to be 7.8GB's (On W7/DS 4.15)

  • duckdoom5duckdoom5 Posts: 11
    edited September 2021

    Ah, the RTX emulation is a very good point. I guess that would take it's fair share of memory.

    I should totally get an RTX card, but I'm a bit too short on money and I'd have to build a complete desktop pc (running on a laptop rn).

     

    I'm gonna assume the issue is caused by the scene using too much VRAM then. I'll try to get a scene optimizer plugin from the store and reduce the load.

    Thanks for the help

    Post edited by duckdoom5 on
  • I'm having a similar issue with my RTX 3070, it starts the render process and after 30 seconds ...nothing, here's what my log says:

    2021-11-28 17:37:41.191 Iray [INFO] - IRAY:RENDER ::   1.10  IRAY   rend progr: CUDA device 0 (NVIDIA GeForce RTX 3070): Processing scene...
    2021-11-28 17:37:41.262 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Geometry memory consumption: 201.682 MiB (device 0), 0.000 B (host)
    2021-11-28 17:38:18.424 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Texture memory consumption: 5.746 GiB for 361 bitmaps (device 0)
    2021-11-28 17:38:18.497 Iray [INFO] - IRAY:RENDER ::   1.12  IRAY   rend info : Importing lights for motion time 0
    2021-11-28 17:38:18.530 Iray [INFO] - IRAY:RENDER ::   1.12  IRAY   rend info : Initializing light hierarchy.
    2021-11-28 17:38:18.862 Iray [INFO] - IRAY:RENDER ::   1.12  IRAY   rend info : Light hierarchy initialization took 0.244s
    2021-11-28 17:38:18.882 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Lights memory consumption: 1.947 MiB (device 0)
    2021-11-28 17:38:18.902 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Material measurement memory consumption: 0.000 B (GPU)
    2021-11-28 17:38:19.587 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : PTX code (3.993 KiB) for SM 8.0 generated in 0.677s
    2021-11-28 17:38:19.596 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Materials memory consumption: 1.499 MiB (GPU)
    2021-11-28 17:38:19.972 Iray [INFO] - IRAY:RENDER ::   1.12  IRAY   rend info : JIT-linking wavefront kernel in 0.136s
    2021-11-28 17:38:19.973 Iray [INFO] - IRAY:RENDER ::   1.10  IRAY   rend info : CUDA device 0 (NVIDIA GeForce RTX 3070): Scene processed in 38.773s
    2021-11-28 17:38:20.083 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.10  IRAY   rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Not enough memory for kernel launches (0.000 B (464.133 MiB) required, 0.000 B available). Cannot allocate framebuffer.
    2021-11-28 17:38:20.098 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.10  IRAY   rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Failed to setup device frame buffer
    2021-11-28 17:38:20.098 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.10  IRAY   rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Device failed while rendering
    2021-11-28 17:38:20.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER ::   1.10  IRAY   rend warn : CUDA device 0 (NVIDIA GeForce RTX 3070) ran out of memory and is temporarily unavailable for rendering.
    2021-11-28 17:38:20.106 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER ::   1.10  IRAY   rend warn : All available GPUs failed.
    2021-11-28 17:38:20.107 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.10  IRAY   rend error: Fallback to CPU not allowed.
    2021-11-28 17:38:20.107 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER ::   1.10  IRAY   rend error: All workers failed: aborting render
    2021-11-28 17:38:20.117 Iray Render error: Internal rendering error.

  • PerttiAPerttiA Posts: 4,503

    commo81 said:

    I'm having a similar issue with my RTX 3070, it starts the render process and after 30 seconds ...nothing, here's what my log says:

    2021-11-28 17:38:18.424 Iray [VERBOSE] - IRAY:RENDER ::   1.12  IRAY   rend stat : Texture memory consumption: 5.746 GiB for 361 bitmaps (device 0)

    That is the problem, you are running out of VRAM.

    On a 8GB GPU, if geometry+textures+working space take more than ~5.7GiB's, it will fall back to CPU (if allowed), you are using that just on the textures.

Sign In or Register to comment.