CUDA doesn't have enough memory to launch the second time
duckdoom5
Posts: 11
So when using Iray to render my scene I get this weird issue that I've not seen many people talk about.
I had this problem before where it was very slow and I found out it was falling back on the CPU. I turned it off so I can now see when it tries to fallback.
So I can render everything fine once, but then when I try to start the next render session it seems like Iray / Daz didn't clear the CUDA launch kernel memory.
Restarting daz doesn't help. The only things that work are singing out/restarting my pc or reinstalling the graphics drivers. It's quite annoying to have to restart my pc every time I need to render for a second time.
Any ideas? Maybe someone knows a way to reset the memory?
Also, it doesn't appear to be VRAM. I have a bit less then 4 GB's still available (see attached ss), and if I add up the memory in the logs it's about 2.5 GB's
2021-09-23 12:22:36.772 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Initializing OptiX Prime for CUDA device 0
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering with 1 device(s):
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : CUDA device 0 (NVIDIA GeForce GTX 1070)
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering...
2021-09-23 12:22:36.777 Iray [INFO] - IRAY:RENDER :: 1.3 IRAY rend progr: CUDA device 0 (NVIDIA GeForce GTX 1070): Processing scene...
2021-09-23 12:22:37.156 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : Geometry memory consumption: 1.003 GiB (device 0), 0.000 B (host)
2021-09-23 12:22:56.602 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : Texture memory consumption: 1.361 GiB for 237 bitmaps (device 0)
2021-09-23 12:22:56.616 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend info : Importing lights for motion time 0
2021-09-23 12:22:56.617 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend info : Initializing light hierarchy.
2021-09-23 12:22:56.778 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend info : Light hierarchy initialization took 0.161s
2021-09-23 12:22:56.779 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : Lights memory consumption: 1.365 MiB (device 0)
2021-09-23 12:22:56.906 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : Material measurement memory consumption: 0.000 B (GPU)
2021-09-23 12:22:56.986 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : PTX code (2.704 KiB) for SM 6.1 generated in 0.046s
2021-09-23 12:22:56.987 Iray [VERBOSE] - IRAY:RENDER :: 1.8 IRAY rend stat : Materials memory consumption: 39.792 MiB (GPU)
2021-09-23 12:22:57.158 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend info : JIT-linking wavefront kernel in 0.073s
2021-09-23 12:22:57.191 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend info : JIT-linking mega kernel in 0.033s
2021-09-23 12:22:57.191 Iray [INFO] - IRAY:RENDER :: 1.3 IRAY rend info : CUDA device 0 (NVIDIA GeForce GTX 1070): Scene processed in 20.414s
2021-09-23 12:22:57.192 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.3 IRAY rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Not enough memory for kernel launches (53.762 MiB (304.762 MiB) required, 0.000 B available). Cannot allocate framebuffer.
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.3 IRAY rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Failed to setup device frame buffer
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.3 IRAY rend error: CUDA device 0 (NVIDIA GeForce GTX 1070): Device failed while rendering
2021-09-23 12:22:57.193 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER :: 1.3 IRAY rend warn : CUDA device 0 (NVIDIA GeForce GTX 1070) ran out of memory and is temporarily unavailable for rendering.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER :: 1.3 IRAY rend warn : All available GPUs failed.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.3 IRAY rend error: Fallback to CPU not allowed.
2021-09-23 12:22:57.197 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.3 IRAY rend error: All workers failed: aborting render
2021-09-23 12:22:57.198 Iray Render error: Internal rendering error.
2021-09-23 12:22:57.237 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Device statistics:
2021-09-23 12:22:57.237 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : CUDA device 0 (NVIDIA GeForce GTX 1070): 0 iterations, 20.421s init, 0.005s render
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2023 Daz Productions Inc. All Rights Reserved.
Comments
What are the specs of your computer and how much VRAM does the GTX 1070 have?
I've attached a screenshot of my specs
I don't know why, but it's not attaching my second ss.
Edit: Got it attached, I guess it had issues with an & in the image file name
Ok, what other programs do you have open and doing something at the same time?
At which stage did yo take the screenshot of Task manager - 4.4GB's of 8GB's used at that point, not much left for DS.
In an other screenshot, it's saying 29GB's of Page file used...
You are also using W11, which can cause unseen glitches
Thanks for trying to help. Windows 11 is definitely not the issue here as I had this before on windows 10. Also on less VRAM taxing scenes.
Though after further investigation I found that when it actually is able to render the scene, it's using close to maximum amount of VRAM. (In other words, it's using way more memory then reported in the logs)
I'm guessing that DAZ/ Iray doesn't release all the VRAM after a render, causing it to be insufficient when I render the second time.
To answer your question, I took that ss of Task Manager right after I got the message it failed to start the render. I also only have daz open, no other programs
The OS, DS and the baseload of the scene are taking their share of the VRAM and using a GTX, means that some additional VRAM is used for emulating the RTX functions in software.
With an RTX 2070 Super (8GB), According to the log, about the maximum the geometry, textures and working space can have is 5.7GB's for the scene still to render with the GPU - The GPU-Z reports total VRAM usage at this point to be 7.8GB's (On W7/DS 4.15)
Ah, the RTX emulation is a very good point. I guess that would take it's fair share of memory.
I should totally get an RTX card, but I'm a bit too short on money and I'd have to build a complete desktop pc (running on a laptop rn).
I'm gonna assume the issue is caused by the scene using too much VRAM then. I'll try to get a scene optimizer plugin from the store and reduce the load.
Thanks for the help
I'm having a similar issue with my RTX 3070, it starts the render process and after 30 seconds ...nothing, here's what my log says:
2021-11-28 17:37:41.191 Iray [INFO] - IRAY:RENDER :: 1.10 IRAY rend progr: CUDA device 0 (NVIDIA GeForce RTX 3070): Processing scene...
2021-11-28 17:37:41.262 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : Geometry memory consumption: 201.682 MiB (device 0), 0.000 B (host)
2021-11-28 17:38:18.424 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : Texture memory consumption: 5.746 GiB for 361 bitmaps (device 0)
2021-11-28 17:38:18.497 Iray [INFO] - IRAY:RENDER :: 1.12 IRAY rend info : Importing lights for motion time 0
2021-11-28 17:38:18.530 Iray [INFO] - IRAY:RENDER :: 1.12 IRAY rend info : Initializing light hierarchy.
2021-11-28 17:38:18.862 Iray [INFO] - IRAY:RENDER :: 1.12 IRAY rend info : Light hierarchy initialization took 0.244s
2021-11-28 17:38:18.882 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : Lights memory consumption: 1.947 MiB (device 0)
2021-11-28 17:38:18.902 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : Material measurement memory consumption: 0.000 B (GPU)
2021-11-28 17:38:19.587 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : PTX code (3.993 KiB) for SM 8.0 generated in 0.677s
2021-11-28 17:38:19.596 Iray [VERBOSE] - IRAY:RENDER :: 1.12 IRAY rend stat : Materials memory consumption: 1.499 MiB (GPU)
2021-11-28 17:38:19.972 Iray [INFO] - IRAY:RENDER :: 1.12 IRAY rend info : JIT-linking wavefront kernel in 0.136s
2021-11-28 17:38:19.973 Iray [INFO] - IRAY:RENDER :: 1.10 IRAY rend info : CUDA device 0 (NVIDIA GeForce RTX 3070): Scene processed in 38.773s
2021-11-28 17:38:20.083 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Not enough memory for kernel launches (0.000 B (464.133 MiB) required, 0.000 B available). Cannot allocate framebuffer.
2021-11-28 17:38:20.098 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Failed to setup device frame buffer
2021-11-28 17:38:20.098 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: CUDA device 0 (NVIDIA GeForce RTX 3070): Device failed while rendering
2021-11-28 17:38:20.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER :: 1.10 IRAY rend warn : CUDA device 0 (NVIDIA GeForce RTX 3070) ran out of memory and is temporarily unavailable for rendering.
2021-11-28 17:38:20.106 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [WARNING] - IRAY:RENDER :: 1.10 IRAY rend warn : All available GPUs failed.
2021-11-28 17:38:20.107 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: Fallback to CPU not allowed.
2021-11-28 17:38:20.107 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.10 IRAY rend error: All workers failed: aborting render
2021-11-28 17:38:20.117 Iray Render error: Internal rendering error.
That is the problem, you are running out of VRAM.
On a 8GB GPU, if geometry+textures+working space take more than ~5.7GiB's, it will fall back to CPU (if allowed), you are using that just on the textures.