IRAY Errors in logs (video memory issues) seen in DS 4.10 and NOT DS 4.9
Since installing Daz Studio 4.10 I've noticed the following in my logs.
Context: IRAY render of moderately complex scene (image attached). My rig is a Windows 10 64-bit system with 64 gigs of ram and dual Nvidia GEForce Titan X with 12 gigs VRAM each (max 12 gigs usable). 10.7 gigs in use during the render according to GPU-Z which I always have running to monitor my video cards. I set DAZ Studio to use CPU and video card 1 only so that I could isolate the work being done and memory used to my IRAY render process.
Every iteration that writes a log entry also writes the following:
2017-10-27 17:06:09.511 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.6 IRAY rend error: CUDA device 1 (GeForce GTX TITAN X): May not have enough memory available to execute CUDA kernel (estimated 176.597 MiB).
2017-10-27 17:06:09.512 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.6 IRAY rend error: CUDA device 1 (GeForce GTX TITAN X): out of memory (while allocating memory)
2017-10-27 17:06:09.512 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.6 IRAY rend error: CUDA device 1 (GeForce GTX TITAN X): Failed to allocate 25.8252 MiB
There's over 1 gig available in the video card, and I've never seen these errors before. I rendered this exact scene yesterday using DAZ Studio 4.9 with no errors or warnings in the logs.
I'm thinking something was introduced in v4.10 that might have changed the way DAZ Studio communicates with the video cards during IRAY renders. While this is not a show stopper per se, it is alarming because I often render larger scenes and rely on DAZ Studio to communicate effectively with my cards.
Has anyone else seen this?
Comments
Here's the image I rendered:
Hi, I also noticed some strange entries in the log. When rendering a scene in 4.3 GB on a single 540M/1Gb VRAM video card:
2017-10-29 18:19:19.459 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Initializing local rendering.
2017-10-29 18:19:19.522 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Rendering with 1 device(s):
2017-10-29 18:19:19.522 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : CUDA device 0 (GeForce GT 540M)
2017-10-29 18:19:19.522 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Rendering...
2017-10-29 18:19:19.553 Iray VERBOSE - module:category(IRAY:RENDER): 1.4 IRAY rend progr: CUDA device 0 (GeForce GT 540M): Processing scene...
2017-10-29 18:19:19.615 Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Geometry memory consumption: 17.2855 MiB (device 0), 0 B (host)
2017-10-29 18:23:49.801 Iray INFO - module:category(IRAY:RENDER): 1.2 IRAY rend info : Initializing light hierarchy.
2017-10-29 18:23:52.113 Iray INFO - module:category(IRAY:RENDER): 1.2 IRAY rend info : Light hierarchy initialization took 2.08s
2017-10-29 18:23:53.441 Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Texture memory consumption: 4.28346 GiB (device 0)
2017-10-29 18:23:54.770 Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Material measurement memory consumption: 0 B (GPU)
2017-10-29 18:23:54.770 Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Materials memory consumption: 370.117 KiB (GPU)
2017-10-29 18:23:56.145 Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Lights memory consumption: 196 B (device 0)
2017-10-29 18:23:59.567 Iray INFO - module:category(IRAY:RENDER): 1.4 IRAY rend info : CUDA device 0 (GeForce GT 540M): Scene processed in 279.928s
2017-10-29 18:24:00.129 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.4 IRAY rend error: CUDA device 0 (GeForce GT 540M): out of memory (while allocating memory)
2017-10-29 18:24:00.301 WARNING: dzneuraymgr.cpp(307): Iray ERROR - module:category(IRAY:RENDER): 1.4 IRAY rend error: CUDA device 0 (GeForce GT 540M): Failed to allocate 528.158 MiB
2017-10-29 18:24:00.535 Iray INFO - module:category(IRAY:RENDER): 1.4 IRAY rend info : CUDA device 0 (GeForce GT 540M): Allocated 15.176 MiB for frame buffer
2017-10-29 18:24:03.567 Iray INFO - module:category(IRAY:RENDER): 1.6 IRAY rend info : CUDA device 0 (GeForce GT 540M): Prevent device timeout
2017-10-29 18:24:03.582 Iray INFO - module:category(IRAY:RENDER): 1.2 IRAY rend info : CUDA device 0 (GeForce GT 540M): Prevented device timeout
2017-10-29 18:24:05.379 Iray INFO - module:category(IRAY:RENDER): 1.4 IRAY rend info : Allocating 1 layer frame buffer
2017-10-29 18:24:05.989 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Received update to 00001 iterations after 286.452s.
2017-10-29 18:24:10.255 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Received update to 00002 iterations after 290.721s.
...
2017-10-29 18:34:06.800 Total Rendering Time: 14 minutes 57.25 seconds
...
2017-10-29 18:34:31.379 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Device statistics:
2017-10-29 18:34:31.379 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : CUDA device 0 (GeForce GT 540M): 100 iterations, 282.107s init, 447.121s render
I would like to see some comments from the developers for this unusual behavior. In previous versions of the DS was clearly written that there is a transition to the CPU render. Also in discussions with others, it turned out that such a picture is observed and they have. If there is a shortage of video memory, writes the same errors to the log and continues (or "continues"?) rendering on the video card with an error.
Since 4.10 installation, I've gotten crashes in relatively small setups, as well (running two Titan Xp's). Maybe that's my issue!
I've also been receiving tons of these in the log
2018-01-02 16:06:58.216 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Received update to 02501 iterations after 5500.402s.
2018-01-02 16:06:58.231 Iray INFO - module:category(IRAY:RENDER): 1.10 IRAY rend info : CUDA device 0 (GeForce GTX 1050 Ti): Prevent device timeout
2018-01-02 16:06:58.253 Iray INFO - module:category(IRAY:RENDER): 1.4 IRAY rend info : CUDA device 0 (GeForce GTX 1050 Ti): Prevented device timeout
2018-01-02 16:06:59.885 Iray INFO - module:category(IRAY:RENDER): 1.10 IRAY rend info : CUDA device 0 (GeForce GTX 1050 Ti): Prevent device timeout
2018-01-02 16:06:59.902 Iray INFO - module:category(IRAY:RENDER): 1.4 IRAY rend info : CUDA device 0 (GeForce GTX 1050 Ti): Prevented device timeout
Running Newest nvidia drivers as well as of 12/20/2017