UPdating my Renderfarm
Boojum the brown bunny
Posts: 348
My work REALLY loves me, they really do. ;) My work was cutting loose some "old" Dell 2950 rackmounted servers, and they let me have them. They wouldn't fit in my old rack, so I picked up a new StarTech rack to mount them in. I didn't grab enough rails from work, and work has since gotten rid of the extra rails they had. So I had to buy 5 new server rails on Ebay for about $120. My current 15 amp power strip on the rack is completely unable to keep up with the load, so I have purchased 2 more 15 amp power strips. They should arrive in the next couple of days. Once that is all mounted, I will finish cable tieing my cables to the sides of the racks, mount my Network Storage, and I will be ready to render!
I'll let you all know how it goes. According to my KillaWatz, when not rendering, the machines are drawing 2.7 amps each, at 336 watt. Each machine is dual CPU with between 32gb and 64gb of ram. I have 11 servers, and my goal will be to have 1 server have the full carrara on it as the master for the farm, and then have 10 machines running Rendernode. Each machine is running Windows 10 Pro, 64-bit.
The servers are now in my rack and look like this:
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Now that's a nice setup!
Wow!!
I'm new to network rendering but have collected a couple of dual CPU servers and 6 other computers/laptops, but I just ran into a limitation of some kind, as I should be able to render with 80 render cores, but only 72 render cores are being used, while 8 stand idle. I already have Grid, so I don't understand why I'm hitting a limitation as I didn't think I should hit the ceiling till I went over 100 render cores.
But just a glance at your setup makes me think this new addition to your farm will put you waaay over 100 render cores. Are you worried about any built in limitations Carrara might have that will not take advantage of all that potential rendering power?
The 2950's are either duo core or quad core with up to 2 CPU's so he may not hit your wall.
11x2=22X4=88 Cores
This looks similar the the server room I work in but with lots more Servers.
These are all quad core (with one exception that I will be updating this weekend.) I am also a grid owner (and have been for many, many years) so I am not particularly worried about hitting the limit, but I will post to this thread if I do.. and of any other problems that I run into. Hopefully that will help others who are setting up their Renderfarm.
I know the first time I set up a Renderfarm, I had to copy my Plants folder from the main machine over to all the Rendernodes, or the Rendernodes couldn't find the plants in my scene. I'm not sure if that is the same case now. I also had to copy the extensions folder over or they couldn't use my plugins. To make my life simple, I shared the Carrara folder on each rendernode machine, which let me easily copy things from the main machine.
Boojum
I think maybe that was only for an earlier iteration of Carrara, as I didn't copy the extensions or plant folders over to any of my nodes and they seem to work fine... with the exception that I seem to top out at 72 cores (I suppose I'm actually topping out at 64 render cores if you discount my primary machine). I'm going to try the suggestion though to see if it helps. Here's a pic of me rendering with all nodes, but only getting a mere 72 cores rendering instead of the 80 I should be getting (seems petty to whine about it since I realize I have more render cores now than I'll likely ever need, and I can render with ultimate high settings on anything and still get a decent render time, but still...)
Oh and I did test to see if it was something wrong with that particular node too, turning off the other nodes and rendering with just that node, and it rendered fine, so there's no problem of the primary PC communicating and using that node.
So freaking cool!!! This stuff just fires me right up!!! LOL
BTW... Hi Brown Bunny!!! Haven't seen you in a while! <waves>
I called Daz support today, to report on the problem with Grid, of course, the woman who answered the phone had no idea what Grid even was (I suspect she probably didn't know much about Carrara either) but she opened a ticket after I patiently explained the disparity between the number of CPU's the sales page said were supported vs the 64 render core limitation I was running into, and said the Carrara tech support specialists (of which I'm sure there must be gigantic office buildings filled with thousands of Carrara tech support engineers
) would be getting right back to me within 7 - 24 years with an answer. 
I wonder if it's just a bottleneck issue? I know that my workstation is more than three times faster than my laptop, and yet it can sometimes take a while for it to be added to the render. Just wondering if it would eventually kick on if given more time?
Damn you jonstark! 80 cores! And you, Brown Bunny... 88 cores?!!! I'm so jealous!
I don't think it's possible that it could be, all the nodes are very close to the same power and Intel-generation and all kick on at the same time. The laptop that is being left out is just as fast in other tests where I've excluded the other nodes to kick on and start contributing, and I waited a good long time for it to start contributing, but nothing happened (I could also see nothing was happening on the laptop in question either).
Oh... do the nodes actually 'show' something happening while they're working? I never looked
It does depend on the scene and settings, but for example, with GI it will show the light calculations progress bar on each of the nodes, or if SSS needs to be calculated, things like that.
Cool
Are they connected with a Gigabit Router and are the network cards Gigabit as they me only be 100 MB and that may bottle neck you when they are communicating with the master..
I actually don't know the answer to that, I may have to google 'Gigabit Router' for more info, lol. I've got a wireless linksys router with 4 ports and an ethernet switch with 4 more ports, and 2 of the laptops are only connected wirelessly. I don't really see why there would be a bottleneck in the router though; I have other non-network computers that are connected through the router to the internet and able to stream movies at the same time with no issues, so I can't imagine there's an information bandwidth issue slowing things down.
Just looked up my router model number and it is a Gigabit router, so I guess the problem isn't in any port speed throttling.
I did get an email from Daz support where they essentially copied and pasted the sales copy on the Carrara grid page. In fairness, it was very clear that the person I spoke to on the phone didn't accurately represent the issue I was running into, so I've written the guy back clarifying the problem. Maybe I'll hear something back tomorrow.
So here is some things I have learned about Renderfarms.
1. do not render over Wifi. First, because Wifi is shared bandwidth (two machines get half the throughput, four machines get a quarter the throughput). Second, because Wifi is best effort and I have had renders die because important parts never got sent.
2. Make sure you are using at LEAST Cat5e cables. If you are using Cat5 then they won't support gigabit speeds.
I'll be doing my test renders this weekend! YAY!
Boojum
When you're done testing, send that rack system to my house. I want to try something for a few years... then I'll give it right back! ;)
Ok! So I have some initial numbers. I have 7 working servers in the rack right now... the other 4 need a bit of work that I haven't had time to do yet. (Still, free servers is really cool!)
I rendered a Rolling Hills picture in 1080p high def resolution. On my i7 Main machine it took 4 hours 5 minutes to render. On my main machine AND the 7 renderfarm machines it took 42 minutes to render. Not a bad speed bump, and it should get better when I get the other 4 machines working.
Final Image (click through for full res):
RenderFarm in progress, with LOTS of N's! (click through for full res):
That's awesome Boojum, cutting render time by more than 80% isn't bad at all, and will of course be even lower render times when you get those other 4 servers into the mix. It's a thrill to see the screen fill up with all those grey render buckets
Daz support is advising me to try making one of my servers the main machine and see if I hit the 64 core ceiling for the nodes or not, and I'm going to give it a try, but the thing is my servers are both 14 render cores, so if I make one of them the main machine, those 16 cores don't count towards the Grid maximum for the nodes. I'll have to borrow my brother's laptop so I can get above 64 render cores to see if it can exceed the limit I hit previously or not. Going to also make sure this time all is wired over ethernet cables of the latest gen so that I eliminate all other possible factors, since Daz support advised they were unable to replicate my problem (I assume this means they tested and were able to exceed 64 render cores without hitting any ceiling or having any of the nodes go unused).
I counted 68 total cores (60 rendernode cores and 8 main machine cores). I will hopefully get the extra machines running this weekend, which should exceed your limit. For the record, I also have grid on my main machine.
Jeff
Wow Boojum, that is incredible! And I was walking on air when I made the leap from a single core processor to my current single quad core...You all are awseome!!
I remember going from a single core laptop to a dual core laptop and how awesome that was indeed, then to an i5 and having 4 render cores, and much more recently getting an i7 laptop and seeing 8 (!) brightly colored renderbuckets racing across the screen
Euphoria!
And nowadays with a cheap purchase on ebay anyone can add a render node very inexpensively to put tons more render buckets on the screen and decrease the render time. We live in age of wonders, indeed!
Arrrrrrggggg.. They got me.. the not all CPU's used for render blues. Carrara calls Cores "CPU's" So I am running Carrara 8.5 Pro with Grid. This should allow me to have up to 100 CPU's all networked together and rendering, and instead I am topping out at 61 Networked CPU's. The pictures I am attaching are clickthroughs so people can see what's going on. I'm about to go open a bug on it and see if someone can help me. First, here is my network log:
Start Rendering:
IP Address: 10.0.0.189
Subnet Mask: 255.255.255.0
Broadcast to: 10.0.0.182
Broadcast to: 10.0.0.131
Broadcast to: 10.0.0.229
Registering Render Node: Cleric (10.0.0.182) 8 CPUs
Registering Render Node: Bard (10.0.0.131) 8 CPUs
Registering Render Node: Thief (10.0.0.229) 4 CPUs
Broadcast to: 10.0.0.181
Broadcast to: 10.0.0.84
Broadcast to: 10.0.0.44
Broadcast to: 10.0.0.148
Broadcast to: 10.0.0.59
Broadcast to: 10.0.0.32
Broadcast to: 10.0.0.203
Registering Render Node: Thug (10.0.0.181) 4 CPUs
Registering Render Node: Wizard (10.0.0.84) 8 CPUs
Registering Render Node: Ranger (10.0.0.44) 8 CPUs
Registering Render Node: Barbarian (10.0.0.148) 8 CPUs
Registering Render Node: Spellsinger (10.0.0.59) 8 CPUs
Registering Render Node: Hunter (10.0.0.32) 8 CPUs
Registering Render Node: Rogue (10.0.0.203) 8 CPUs
Broadcast End render to: 10.0.0.182
Broadcast End render to: 10.0.0.131
Broadcast End render to: 10.0.0.229
Broadcast End render to: 10.0.0.181
Broadcast End render to: 10.0.0.84
Broadcast End render to: 10.0.0.44
Broadcast End render to: 10.0.0.148
Broadcast End render to: 10.0.0.59
Broadcast End render to: 10.0.0.32
Broadcast End render to: 10.0.0.203
Total Nodes: 0
Total CPUs: 72
So I have 72 CPU's as counted by Carrara. Here is my rendering picture, as well as a picture with what the nodes are doing captured. As you can see, one machine isn't even being used... and I should have way more cores showing... like another 11 CPU's (or N's)
Damn, pretty much exactly where I topped out too, I was able to render with 72 cores, counting the main PC, 64 cores on the various nodes, with one machine and 8 cores going unused.
If it helps when you raise the bug report, the ticket I raised for this issue is Request #228872
Sometime before the weekend is done I'll put together the time to make one of my dual-xeon servers the main PC as was suggested by the Daz tech support, to see if that makes any difference or not, I admit I'm not terribly hopeful though, especially after seeing it looks like you ran into the same wall I did.
Oh, I also have to say I love the way you named your nodes after various RPG classes, very cool
*laugh* What can I say.. roleplaying games are in my blood, and why I got started in 3d renders to begin with.
For right now, I'm just not going to start the two 4 core systems and see how it does. I'm rendering an animation right now.
LOL! That is SO Cool!!! RPGs got me into all of this too! :)
Ok.. so I now have 9 machines rendering an animation. Interestingly enough, all 9 machines, with 8 cpu's each, plus my main machine with 8 cpu's, are all working. So that's better than what I had before. I'm betting those two dual core systems didn't play well with the quad core systems. So now I have 72 "cpu's" plus 8 cpu's for my main machine, for 80 CPU's. I think my next step will be to add in another i7 system and see if I get 80 plus 8.... which would mean that the limit is on how many cpu's can work on a single image but all can still work on an animation.
What games got you started? I'm currently in 10 play by post RPG's on my forums, and 2 live games (one Pathfinder, one AD&D 2nd edition)
Boojum
AD&D got me started, though years before that, I was asked to be the featured artist in a D&D club's magazine, so I did a bunch of fantasy art for the whole thing - even illustrations bordering the page numbers... was really fun, yet I had no idea what D&D was all about yet! LOL
So when I started buying my own books, AD&D 2nd Edition was brand new, so I bought that. I'm glad I did too! 1st edition required both Player's Handbbok and the Dungeon Master's Guide to build a complete character! But the DMG sure had some cool charts and tables that 2nd edition lacks. I have a whole bunch of 2nd edition stuff, including the DVD of the complete works, the complete collection of 1990's AD&D 2nd collector's cards, and the Wizard and Cleric card sets that came out back then - and really close to all of the Forgotten Realms of 2nd edition.
Neverwinter Nights (first edition) really stole my heart! By then I had moved to where I live now and don't have anyone to RPG with. Playing NWN over the web is super-awesome with the Dungeon Master tools and everything! I was part of the CCP (Community Content Project) and CRAP (Classic Roleplay Adaptation Project) and was able to co-DM a wonderful Star Wars campaign, but playing has always been my favorite part. I used to DM back in the day... but even then I'd rather play than DM.
Never tried Pathfinder, but it certainly looks amazing! Might get that one of these years.
I really liked the d20 Modern stuff. Never played it but I have a bunch of the books.
Recently through GOG, I got Shadowrun, three of them (the only 3 I know of) and just love them! I'll keep playing through them - so fun!
Drat, I take it back. Only 8 machines are rendering, and the 9th is sitting idle. This means I topped out at 64 CPU's and 8 dual CPU Machines. No where near what I was hoping for.
Sounds like you have a lot of fun playing! I've been in my saturday game for over 10 years now, and still having a lot of fun!
Boojum
So this was the weirdest thing. I finally tested it running Carrara from one of my 16 core servers, and while I still hit the 64 core limit for what the Nodes could do, It actually wasn't using 3 of my laptop nodes at all, which means that it had more render cores for the Nodes than it should have had. I still got up to 80 render cores counting the main machine, but since 3 of the Nodes were sitting idle, it should have had less render cores than it was showing, check out the pics: